Extracting entity relations with newly trained (from pretrained) named entity recognizer

[GarethEgerton]

I've posted my code and question on stackoverflow however didn't manage to get a response, not sure if this is a bug?

https://stackoverflow.com/questions/60621365/spacy-extract-named-entity-relations-from-trained-model

I've trained a model (using "en_core_web_sm" as base model transfer learning) to detect a new entity called "CASES". My new model successfully detects if the new entity is in a sentence, and remains able to detect other named entities.

However when using the new model to extract entity relations using the example code on spacy website. https://spacy.io/usage/examples

The new model gives no results... whilst the old base "en_core_web_sm" works as intended.

I'm not sure why this is, has this ability been deactivated somehow in the new model?

• spaCy version: 2.1.3
• Platform: Windows-10-10.0.18362-SP0
• Python version: 3.7.6

import numpy as np
import pandas as pd
import os
import random
from IPython.core.display import HTML
import matplotlib.pyplot as plt
import numpy as np
from pandas.api.types import CategoricalDtype
import datefinder
from datetime import datetime, timedelta
from make_dataset import articles_database
from TFIDF import spacy_tfidf 
import datefinder

import spacy
nlp = spacy.load("en_core_web_sm")

from __future__ import unicode_literals, print_function
from spacy.lang.en import English

import plac
import random
from pathlib import Path
import spacy
from spacy.util import minibatch, compounding

# new entity label
LABEL = "CASES"

# the raw training data (i've included 3 examples, but am actually training with 1000)

TRAIN_DATA = [('1996 - Global Cholera Update 22 January 1996 Disease Outbreak Reported Cap Verde : Cholera has been reported every week since\nNovember 1994.',
  {'entities': [(0, 4, 'DATE'),
    (29, 44, 'DATE'),
    (109, 119, 'DATE'),
    (126, 139, 'DATE')]}),
 ('By 5 November 1995 at total of 12,344 with 245\ndeaths have been notified.',
  {'entities': [(3, 18, 'DATE'), (31, 37, 'CARDINAL'), (43, 46, 'CARDINAL')]}),
 ('Of these, 879 cases with 4 deaths\nwere reported for the period 9 October to 5 November 1995.',
  {'entities': [(14, 19, 'CASES'),
    (10, 13, 'CARDINAL'),
    (25, 26, 'CARDINAL'),
    (63, 64, 'CARDINAL'),
    (76, 77, 'CARDINAL'),
    (78, 91, 'DATE')]}),
 ('Control efforts have not succeeded in preventing the spread of the\nepidemic and when cases were detected on Sao Nicolau and Sal\nIslands in the period 9 October to 5 November all nine inhabited\nislands of Cap Verde had become infected.',
  {'entities': [(85, 90, 'CASES'),
    (108, 119, 'GPE'),
    (124, 135, 'GPE'),
    (150, 151, 'CARDINAL'),
    (163, 164, 'CARDINAL'),
    (165, 173, 'DATE'),
    (178, 182, 'CARDINAL'),
    (204, 213, 'PERSON')]}),
 ('The last cholera\nepidemic in Cap Verde occurred in 1979.',
  {'entities': [(29, 38, 'PERSON'), (51, 55, 'DATE')]})

# COPY AND PASTED from https://spacy.io/usage/examples

def main(model="en_core_web_sm", new_model_name="cases", output_dir='data3', n_iter=30):
    """Set up the pipeline and entity recognizer, and train the new entity."""
    random.seed(0)
    if model is not None:
        nlp = spacy.load(model)  # load existing spaCy model
        print("Loaded model '%s'" % model)
    else:
        nlp = spacy.blank("en")  # create blank Language class
        print("Created blank 'en' model")
    # Add entity recognizer to model if it's not in the pipeline
    # nlp.create_pipe works for built-ins that are registered with spaCy
    if "ner" not in nlp.pipe_names:
        ner = nlp.create_pipe("ner")
        nlp.add_pipe(ner)
    # otherwise, get it, so we can add labels to it
    else:
        ner = nlp.get_pipe("ner")

    ner.add_label(LABEL)  # add new entity label to entity recognizer
    # Adding extraneous labels shouldn't mess anything up
    if model is None:
        optimizer = nlp.begin_training()
    else:
        optimizer = nlp.resume_training()
    move_names = list(ner.move_names)
    # get names of other pipes to disable them during training
    pipe_exceptions = ["ner", "trf_wordpiecer", "trf_tok2vec"]
    other_pipes = [pipe for pipe in nlp.pipe_names if pipe not in pipe_exceptions]
    with nlp.disable_pipes(*other_pipes):  # only train NER
        sizes = compounding(1.0, 4.0, 1.001)
        # batch up the examples using spaCy's minibatch
        for itn in range(n_iter):
            random.shuffle(TRAIN_DATA)
            batches = minibatch(TRAIN_DATA, size=sizes)
            losses = {}
            for batch in batches:
                texts, annotations = zip(*batch)
                nlp.update(texts, annotations, sgd=optimizer, drop=0.35, losses=losses)
            print("Losses", losses)

    # test the trained model   
    
    test_text = "There were 100 confirmed cases?"
    doc = nlp(test_text)
    print("Entities in '%s'" % test_text)F
    for ent in doc.ents:
        print(ent.label_, ent.text)

    # save model to output directory
    if output_dir is not None:
        output_dir = Path(output_dir)
        if not output_dir.exists():
            output_dir.mkdir()
        nlp.meta["name"] = new_model_name  # rename model
        nlp.to_disk(output_dir)
        print("Saved model to", output_dir)

        # test the saved model
        print("Loading from", output_dir)
        nlp2 = spacy.load(output_dir)
        # Check the classes have loaded back consistently
        assert nlp2.get_pipe("ner").move_names == move_names
        doc2 = nlp2(test_text)
        for ent in doc2.ents:
            print(ent.label_, ent.text)
main()


nlp=spacy.load("data3")
test_text = 'Of these, 879 cases with 4 deaths were reported for the period 9 October to 5 November 1995. John was infected. It cost $500'
doc = nlp3(test_text)
print("Entities in '%s'" % test_text)
for ent in doc.ents:
    print(ent.label_, ent.text)

#RESULTS - correctly identifies old and new entities
#Entities in 'Of these, 879 cases with 4 deaths were reported for the period 9 October to 5 #November 1995. John was infected. It cost $500'
#CARDINAL 879
#CASES cases
#CARDINAL 4
#CARDINAL 9
#CARDINAL 5
#CARDINAL $500


#### COPY AND PASTED FROM https://spacy.io/usage/examples but using my new trained model ####'data3'

import plac
import spacy

TEXTS = [
    "Net income was $9.4 million compared to the prior year of $2.7 million. I have 100,000 cases",
    "Revenue exceeded twelve billion dollars, with a loss of $1b.",
]

def main(model="data3"):
    nlp = spacy.load(model)
    print("Loaded model '%s'" % model)
    print("Processing %d texts" % len(TEXTS))

    for text in TEXTS:
        doc = nlp(text)
        relations = extract_currency_relations(doc)
        for r1, r2 in relations:
            print("{:<10}\t{}\t{}".format(r1.text, r2.ent_type_, r2.text))

def filter_spans(spans):
    # Filter a sequence of spans so they don't contain overlaps
    # For spaCy 2.1.4+: this function is available as spacy.util.filter_spans()
    get_sort_key = lambda span: (span.end - span.start, -span.start)
    sorted_spans = sorted(spans, key=get_sort_key, reverse=True)
    result = []
    seen_tokens = set()
    for span in sorted_spans:
        # Check for end - 1 here because boundaries are inclusive
        if span.start not in seen_tokens and span.end - 1 not in seen_tokens:
            result.append(span)
        seen_tokens.update(range(span.start, span.end))
    result = sorted(result, key=lambda span: span.start)
    return result

def extract_currency_relations(doc):
    # Merge entities and noun chunks into one token
    spans = list(doc.ents) + list(doc.noun_chunks)
    spans = filter_spans(spans)
    with doc.retokenize() as retokenizer:
        for span in spans:
            retokenizer.merge(span)

    relations = []
    for money in filter(lambda w: w.ent_type_ == "MONEY", doc):
        if money.dep_ in ("attr", "dobj"):
            subject = [w for w in money.head.lefts if w.dep_ == "nsubj"]
            if subject:
                subject = subject[0]
                relations.append((subject, money))
        elif money.dep_ == "pobj" and money.head.dep_ == "prep":
            relations.append((money.head.head, money))
    return relations

main()

    # Expected output:
    # Net income      MONEY   $9.4 million
    # the prior year  MONEY   $2.7 million
    # Revenue         MONEY   twelve billion dollars
    # a loss          MONEY   1b

    # Actual output:
    # Loaded model 'data3'
    # Processing 3 texts

[svlandeg]

Hm, odd. I can't immediately think of (or see in your code) a reason why this wouldn't work anymore with your new model.

As a sanity check: in the first part of you code where you're testing the newly trained model, can you show the print out results on the test sentences from the currency texts (i.e change test_text to "Net income was..." etc) ? Maybe the set of predicted MONEY entities are not the same?

And then check those results again after loading the model, right before you try to get the currency relations? So we can figure out where this goes wrong.

[svlandeg]

PS: also appreciate you posting on SO first! I did see it there, but as the post was quite long, I didn't have time to look into the details at the time

[GarethEgerton]

Thanks for your feedback.

If I use updated trained model, it can detect the new entity "cases" and cardinal "100,000" however it loses the ability to detect money and date.

When i trained the model, I trained it for thousands of sentences, using the base model en_core_web_sm itself to detect all entities and label them so as to avoid the model "forgetting" the old entities.

[svlandeg]

Ok so it looks like this is an instance of catastrophic forgetting - even though you've tried to avoid it, it may have still crept in.

What I would suggest you to do is:

• Take a bunch of text, run the pretrained model on it, and use these annotations as "silver" evaluation instances. Make sure you have enough examples of each entity type in this set, and output accuracy numbers per entity type.
• Now when you run your evaluation for the pretrained model, it should have "100%" accuracy for each entity type
• Start training your model with your new cases, and run the evaluation inbetween the epochs. Now you'll get a good idea which entity types start suffering while you're retraining. You'll need to make sure you feed in more of these cases during training to avoid the catastrophic forgetting problem.

Because there's not really a bug or action item on our end, I'll close this issue, but feel free to follow up with your results and let me know if you run into further questions / problems.