Differing NER Results when Disabling Pipeline Components

[cjuracek]

Hello! I am interested in performing ner using the en_core_web_sm model. The documentation states "For the non-transformer models, the ner component is independent, so you can disable everything else". Here However, I belive results during testing show otherwise:

from collections import Counter

model = spacy.load('en_core_web_sm')

# Full pipeline
print(f'Running model with: {model.pipe_names}')
docs_full = list(model.pipe(data))
    
# NER only
with model.select_pipes(enable=['ner']):
    print(f'Running model with: {model.pipe_names}')
    docs_ner = list(model.pipe(data))

# Defined this way because I initially thought the entities could be the same up to ordering
def are_ents_equal(ents1, ents2):
    ents1 = [x.text for x in ents1]
    ents2 = [x.text for x in ents2]
    return Counter(ents1) == Counter(ents2)

[are_ents_equal(doc1.ents, doc2.ents) for doc1, doc2 in zip(docs_full, docs_ner)] 

And I only find this to be true 75% of the time over 100 data observations. For reference, the spaCy version is 3.1.3 and the model version is 3.1.0. Any advice?

[adrianeboyd]

Hi, the difference is probably related to sentence boundaries. The ner component doesn't predict entities that cross sentence boundaries, so if you leave out the parser, which sets sentence boundaries, the exact same ner component can make different predictions because the input docs are different. You might be able to check and see that the docs_ner cases have more entities with periods in them like Some Company. Group or similar?

[cjuracek]

Good intuition - I have found this to be the case on a cursory check of the difference in entities. If this is the case, I would propose the documentation be updated, as it explicitly states the parser can be disabled (see: "Disable everything except NER" in the link above). I interpreted this as "you may turn this component off and achieve identical performance", but perhaps that's not what is meant.

[cjuracek]

Update: enabling the parser in addition to the ner still does not result in the same entities. The entities are still the same 75% of the time.

[cjuracek]

Update #2: Enabling the tok2vec and parser give identical entity results as the full pipeline (but not exclusively tok2vec or parser). This is curious to me, as the docs state the ner component "is independent with its own internal tok2vec layer."

[adrianeboyd]

The parser doesn't have its own tok2vec component, so it requires tok2vec or you get nonsense annotation, often something like every token attached to the first word (so no or very few sentence boundaries). You need tok2vec+parser+ner for this check.

The sentence boundary interaction definitely isn't documented well enough, though, so this is pretty confusing, and if I were reading the docs I'd have the same expectation that you did.