coref debugging?

[afriedman412]

Hi --
I'm using the out-of-the-box coreferencing model for a project and I'm trying to work out some of the errors in its predictions. The most common issue is what I called the "odd man out" problem, where the majority of one cluster will refer to one person, but also include a couple references to another.

For example, I have "Robert Whittle" and "Rosie Rivera" in one article, and the relevant clusters look like this:

clusters[3]:
[the father of the man who was killed, his, Robert Whittle, Jason Robert Whittle, 26, of Riverton, Jason Whittle, Whittle, Robert Whittle, his, He, his, He, his, Whittle, Whittle, his, his, he, Whittle, his, he, his, he, Whittle, his, he, Jason Whittle, his, Whittle, his, Whittle, his, the father, He, Rivera, Rivera, she]

clusters[13]:
[Salt Lake County Sheriff Rosie Rivera, Rivera]

The last three spans (Rivera, Rivera, she) shouldn't be in cluster 3, and would ideally be in cluster 13 (but that's less important).

There is nothing fancy in the code itself:

import spacy

coref_nlp = spacy.load("en_coreference_web_trf")
t = open("path/to/file.txt").read()
doc = coref_nlp(t)

How would I go about looking under the hood to get some insight into how the model made its decisions? (Any other debugging tips would be appreciated as well, but I'm just trying to figure out where to start!)

[vinbo8]

Hey, this is a great question, though it is a bit outside the realm of what you can achieve with SpaCy. There's a lot of research on interpreting/explaining deep learning model decisions, and it's far from being a settled science, unfortunately. One thing you could look at is the actual features being generated for each noun phrase, but it's probably not going to be very helpful - these models typically have far too many parameters for us to be able to neatly explain their decisions as a set of rules.

[afriedman412]

right -- appreciate this ... i guess the easier question is is there anything I can tweak to tighten up the results?

[vinbo8]

You could try training your own model, if you have relevant in-domain annotated data.