spacy spancat pipeline performance improvement above texcat

[paocarvajal1912]

How much has the spanCategorizer improved your models? I am curious. I have been using the textcat for categorizing text with a recall of about 85%. I wonder how much applying a spancategorizer could make a difference. I am trying to predict if a question of a questionnaire will bring confidential (personally identifiable) information (such as name, telephone number, address, social security number, etc.). Some questions may be very long, and then the textcat gets confused. I am expecting that being able to catch key terms should improve the prediction, but I wonder how much improvement others have brought to their models. Many thanks for your answers!

[polm]

Let me link #11663, since it's related.

As mentioned there, using NER/spancat annotations as input to textcat is possible, but not very likely to help. #10470 covers this approach and links to some other Discussions on the issue.

That said, if you do try it we'd love to hear about how it goes!

[paocarvajal1912]

I made the comparison between [tok2vec, spancat], [tok2vec, textcat], and [tok2vec, spancat, textcat]. I did it as a first toy sample, with a small sample size of around 600. Here a discussion about some of the code, and a description of the problem.

When running by itself, Textcat was superior than Spancat (recall of 61% versus 32%). When added to Textcat, Spancat did not improve the recall (58% versus 61%). Recall is what I am interested in since I want to be able to identify data as confidential. In other words, I prefer, for example, to say that a pet name is confidential, when it is not, than to say that a social security number is not confidential when it actually is.

However, if you care about accuracy, Spancat was better than Textcat (91% versus 86%), and Spancat Textcat got 88%. My data was unbalanced, so even though Spancat generated more false negatives, still the accuracy was better.

I would like to understand these results better, so would love if you could answer some questions for me.

1. What explains the differences between Spancat and Textcat. Is it related to the technology behind? or are there other reasons? If you can elaborate a little bit about this.

2. Increasing the sample size would impact results for sure, by my experience with using Textcat. How increasing this size would impact Textcat or Spancat? Would Spancat get more benefit than Textcat? Any criteria/ideas I can use to have a preliminary idea?

3. What are the criteria for deciding whether to use Spancat or Textcat?

Thank you very much,
Paola

[polm]

I'm a little confused about your numbers. Usually the accuracy of textcat and spancat wouldn't be directly comparable, because textcat works on whole documents, and spancat works on segments of text within documents, like NER. Have you configured spancat to classify whole documents, or are you doing something else? It might help if you could provide an example of annotations, even for some made up data.

Spancat is usually considered as an alternative to NER, rather than textcat.