Custom spaCy NER model not making expected predictions

[MisbahKhan789]

Issue

A custom NER model (trained to identify certain code numbers) does not produce predictions in certain documents.
The tables below show text snippets of pairs of two documents in row#1 and row#2 respectively, where row#1 represents text on which model returns correct prediction and row#2 represents another text on which model does not return correct prediction, despite text1 and text2 being very similar.

I have tried multiple variations of these texts from row#3 onwards, in an attempt to pinpoint what piece of text is causing differences in predictions. The issue here is- if the model recognizes a given code as the correct inference in one document, why is it not able to identify another similar looking code number as the correct inference, in another similar document.

Model Inputs and Corresponding Outputs

Example1:

Example2:

Environment

• spaCy version: 3.0.6
• Platform: Linux-5.10.178-162.673.amzn2.x86_64-x86_64-with-glibc2.26
• Python version: 3.10.10

[honnibal]

The issue here is- if the model recognizes a given code as the correct inference in one document, why is it not able to identify another similar looking code number as the correct inference, in another similar document.

Unfortunately there's no satisfying answer to this. The model relies contextual representations that incorporate information from up to four words of context on either side of the target token. The entity recognizer then goes through the words of the document as a state machine, and makes decisions about how to construct the entities based on the prior state and the contextual tokens.

When I'm trying to debug the entity recogniser I basically step through the decisions in different ways to try to look at what the state is at a particular decision. The utilities for that aren't documented, and I wouldn't suggest it's the best approach for you to try to understand the behaviour of your system.

Instead, a good mental model to have is basically that the classifier is sensitive to the context as well as just the phrase itself. If you want to be sure that particular phrases are tagged consistently, you could build rule-based matchers, perhaps by running your existing model over a bunch of text, and extracting out lists of phrases which have been tagged at least once. Optionally you could include a manual review step here before you add them to the matcher rules, using an annotation tool like Prodigy.

Other ways you could try to address this are to look at the training and parameterisation of your custom model. If you update to a more recent version of spaCy, you might find some improvement in accuracy (although this isn't guaranteed). Other general advice includes using word vector representations that are well suited for your domain, and pretraining the contextual representations. If you're using the CPU model this could be done with the spacy pretrain command. If you have a GPU, using spacy-transformers and configuring a transformer model for the pipeline would also likely improve accuracy.

Finally, it's worth noting that your input texts aren't really sentential content. It's not uncommon to use NLP tooling on items like yours, just because it's not regular sentences doesn't mean it's somehow trivial to process. But it's worth keeping in mind that your data isn't like normal text, and so some standard recommendations like always preferring statistical approaches to rule-based approaches might not apply to you. The same can be said for choosing word vectors or transformer models. Performance on your task might be quite different from performance on standard benchmarks.

[MisbahKhan789]

Thank you for the detailed response. However, I don't find a clear answer my question.

For two identical texts with just one character difference, why is the model inference (or behavior) so drastically different?

For example: "Product Description: Potassium Cat No. : J/6625/05" and "Product Description: Potassium Cat No. : J/6625/15" (line 5 and 6 of example 1 in question) have difference of just one character, 0 and 1. What could cause the model to behave differently on these two (almost) identical texts?

[honnibal]

Well 10 and 15 get different vector representations, and so the surrounding phrases will get different token representations. The model is probably not well trained to perform confidently on this type of phrase, and this difference in representation happens to be enough that it works out to a different prediction.

I doubt the explanation will end up being meaningfully interesting: it's just that the maths of the prediction will have different inputs and will work out to a different result.

Here's another way of looking at it. There are plenty of situations where a one character difference should completely change the prediction. The model architecture is designed under the assumption that the output doesn't have a linear relationship to the sequence of characters, because the meaning of text generally doesn't.