KeyError: "[E900] Could not run the full pipeline for evaluation.

[Rithik-Alias]

I was training a RE model and the following error came up.
I was running an NER and RE model together. And the data I used for both had a slight variation. Was the error coming because of that?

**Error: **

================================= train_gpu =================================
Running command: /usr/bin/python3 -m spacy train configs/rel_trf.cfg --output training --paths.train data/train.spacy --paths.dev data/dev.spacy -c ./scripts/custom_functions.py --gpu-id 0
ℹ Saving to output directory: training
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2021-10-27 06:52:47,324] [INFO] Set up nlp object from config
[2021-10-27 06:52:47,336] [INFO] Pipeline: ['transformer', 'relation_extractor']
[2021-10-27 06:52:47,341] [INFO] Created vocabulary
[2021-10-27 06:52:47,342] [INFO] Finished initializing nlp object
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.dense.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[2021-10-27 06:52:51,394] [INFO] Initialized pipeline components: ['transformer', 'relation_extractor']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'relation_extractor']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS RELAT...  REL_MICRO_P  REL_MICRO_R  REL_MICRO_F  SCORE 
---  ------  -------------  -------------  -----------  -----------  -----------  ------
⚠ Aborting and saving the final best model. Encountered exception:
KeyError("[E900] Could not run the full pipeline for evaluation. If you
specified frozen components, make sure they were already initialized and
trained. Full pipeline: ['transformer', 'relation_extractor']")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 281, in evaluate
    scores = nlp.evaluate(dev_corpus(nlp))
  File "/usr/local/lib/python3.7/dist-packages/spacy/language.py", line 1388, in evaluate
    results = scorer.score(examples)
  File "/usr/local/lib/python3.7/dist-packages/spacy/scorer.py", line 135, in score
    scores.update(component.score(examples, **self.cfg))
  File "/content/drive/My Drive/NER_RE_New/RE/rel_component/scripts/rel_pipe.py", line 201, in score
    return score_relations(examples, self.threshold)
  File "/content/drive/My Drive/NER_RE_New/RE/rel_component/scripts/rel_pipe.py", line 211, in score_relations
    gold_labels = [k for (k, v) in gold[key].items() if v == 1.0]
KeyError: (11, 18)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/train.py", line 63, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 226, in train_while_improving
    score, other_scores = evaluate()
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 283, in evaluate
    raise KeyError(Errors.E900.format(pipeline=nlp.pipe_names)) from e
KeyError: "[E900] Could not run the full pipeline for evaluation. If you specified frozen components, make sure they were already initialized and trained. Full pipeline: ['transformer', 'relation_extractor']"

[Rithik-Alias]

``The error got solved!!
A little change in the rel_pipe.py file will rectify this error.
open rel_component\scripts\rel_pipe.py file
Inside it change the following function like this,

[polm]

Glad you found a fix!

Sorry for not replying sooner, but for the record I think what happens is that you have predictions that aren't entities, or maybe entities that aren't in your gold document. The scoring in the relation extractor assumes that all candidates are entities it's previously aware of.