Poor performance of Spacy Model

[turulix]

Hey!

I'm trying to develop a model that can extract various types of features or technical details from product descriptions. Currently, I have approximately 34 different features that I need to extract, and I have a dataset of around 10,000 labeled documents.

The problem I'm facing is that when I train the model, it consistently returns a score of 0 for 33 of the labels, and only performs decently for one of them. I'm completely puzzled as to what could be causing this issue.

While it is true that I have limited examples for a few features, with only around 50 or so, I find it perplexing that even features with over 1000 examples are not being recognized at all. It's becoming increasingly challenging to determine the underlying cause of this problem.

I'm currently using this base config:

# This is an auto-generated partial config. To use it with 'spacy train'
# you can run spacy init fill-config to auto-fill all default settings:
# python -m spacy init fill-config ./base_config.cfg ./config.cfg
[paths]
train = "./train.spacy"
dev = "./dev.spacy"
vectors = null
[system]
gpu_allocator = "pytorch"

[nlp]
lang = "de"
pipeline = ["transformer","parser","spancat"]
batch_size = 4

[components]

[components.transformer]
factory = "transformer"
max_batch_items = 64

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "bert-base-german-cased"
tokenizer_config = {"use_fast": true}

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "sc"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 256

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.spancat.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.spancat.suggester]
@misc = "spacy.ngram_suggester.v1"
sizes = [1,2,3]

[components.parser]
source= "de_dep_news_trf"

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
frozen_components = ["parser"]
annotating_components = ["parser"]

[training.optimizer]
@optimizers = "Adam.v1"

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 5e-5

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 1500
buffer = 256

[initialize]
vectors = ${paths.vectors}

[initialize.before_init]
@callbacks = "mod_html_tokenizer"

where the callback in before_init is:

@spacy.registry.callbacks("mod_html_tokenizer")
def html_tokenizer():
    def _html_tokenizer(nlp):
        infixes = nlp.Defaults.infixes + [r'(<)']
        nlp.tokenizer.infix_finditer = spacy.util.compile_infix_regex(infixes).finditer
        nlp.tokenizer.add_special_case(f"<i>", [{ORTH: f"<i>"}])
        nlp.tokenizer.add_special_case(f"</i>", [{ORTH: f"</i>"}])

    return _html_tokenizer

Maybe the training data itself is at fault?
Heres a small excerpt from the Training data itself: https://haste.turulix.de/osixocisof.json

---

Another issue I'm currently facing, for which I'm unsure about the best solution, is handling cases where I'm only interested in a specific part of a word. For example, consider the word Edelstahlmöbel (which means stainless steel furniture in German). When tokenized, it becomes a single token, but I'm primarily interested in the Edelstahl part, which represents the material it is made out of. However, tokenizing the word at the character level doesn't seem like an optimal approach.

While this is just a side note, the main problem I mentioned earlier is the one I'm primarily seeking assistance with. I'm uncertain if there might be a correlation between these two issues, but any insights or suggestions related to either problem would be greatly appreciated.

---

Thank you in advance for your assistance. Best regards and heartfelt thanks in advance,

turulix

[shadeMe]

Could you tell us how you are preprocessing the training data? The presence of HTML tags in the model's input could be affecting its performance. Additionally, please post the output of the debug data command.

[turulix]

I'm currently not really preprocessing the data, since i assumed it would work like this, as it did for the Material label while testing. I'm just ensuring that the HTML tags themself are a seperate token and not connected to any of the sentences inside them. And since the data is already labeled i'm not sure on how to remove them without breaking the position of the label start and endpoints :/

============================ Data file validation ============================
Some weights of the model checkpoint at bert-base-german-cased were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: de
Training pipeline: transformer, parser, spancat
Frozen components: parser
9362 training docs
2341 evaluation docs
✔ No overlap between training and evaluation data

============================== Vocab & Vectors ==============================
ℹ 1528203 total word(s) in the data (35172 unique)
ℹ No word vectors present in the package

============================ Span Categorization ============================

Spans Key   Labels
---------   ------------------------------
sc          {'Kältemittel', 'Grillfläche', 'Bruttoinhalt', 'Abschließbar', 'Leistung_Ofen', 'Einschübe', 'Kippvorrichtung', 'Zündart', 'Thermische_Leistung', 'Türanschlag', 'Pizza_Größe', 'Anzahl_Backkammern', 'Klimaklasse', 'Material', 'Reinigungssystem', 'Türart'
, 'Lüftungsmotor', 'Beleuchtung', 'Korbmaß', 'Farbe', 'Pizzen_je_Backkammer', 'Backkammer', 'Ausstattung_Spülmaschinen', 'Beckengröße', 'Anzahl_Spülbecken', 'Anzahl_Schubladen', 'Kühlung', 'Anschluß', 'Gehäuse', 'Aufstellart', 'Ausstattung_Kombidämpfer', 'Leistung_Kochstellen', 'Anzahl_Kochstellen', 'Zündart_Gasherd', 'Bautiefe'}

⚠ Low number of examples for label 'Abschließbar' in key 'sc' (12)
⚠ Low number of examples for label 'Ausstattung_Spülmaschinen' in key 'sc' (18)
⚠ Low number of examples for label 'Ausstattung_Kombidämpfer' in key 'sc' (40)
⚠ Low number of examples for label 'Reinigungssystem' in key 'sc' (48)
⚠ Low number of examples for label 'Anschluß' in key 'sc' (15)
⚠ Low number of examples for label 'Kühlung' in key 'sc' (43)
⚠ Low number of examples for label 'Klimaklasse' in key 'sc' (37)
⚠ Low number of examples for label 'Kältemittel' in key 'sc' (26)
⚠ Low number of examples for label 'Pizza_Größe' in key 'sc' (28)
⚠ Low number of examples for label 'Beckengröße' in key 'sc' (10)
⚠ Low number of examples for label 'Anzahl_Kochstellen' in key 'sc' (25)
⚠ Low number of examples for label 'Türanschlag' in key 'sc' (9)
⚠ Low number of examples for label 'Kippvorrichtung' in key 'sc' (22)
⚠ Low number of examples for label 'Backkammer' in key 'sc' (17)
⚠ Low number of examples for label 'Pizzen_je_Backkammer' in key 'sc' (13)
⚠ Low number of examples for label 'Grillfläche' in key 'sc' (6)
⚠ Low number of examples for label 'Bruttoinhalt' in key 'sc' (5)
⚠ Low number of examples for label 'Anzahl_Backkammern' in key 'sc' (4)
⚠ Low number of examples for label 'Bautiefe' in key 'sc' (3)
⚠ Low number of examples for label 'Zündart' in key 'sc' (4)
ℹ Span characteristics for spans_key 'sc'
ℹ SD = Span Distinctiveness, BD = Boundary Distinctiveness

Span Type                   Length      SD     BD      N
-------------------------   ------   -----   ----   ----
Beleuchtung                   1.10    6.36   1.26   1199
Material                      1.05    4.45   1.73   7540
Aufstellart                   1.01    7.55   1.44    367
Leistung_Kochstellen          2.23    4.39   1.64    447
Korbmaß                       3.47    4.66   1.98    177
Anzahl_Schubladen             1.10    4.84   3.92    122
Farbe                         1.00    6.53   1.87   1188
Thermische_Leistung           1.98    4.59   1.69   1012
Anzahl_Spülbecken             1.05    5.04   4.52    181
Zündart_Gasherd               1.06    8.27   1.33    201
Leistung_Ofen                 1.94    5.24   1.88    223
Gehäuse                       1.04    7.98   2.58     82
Abschließbar                  1.10    8.49   3.29     12
Ausstattung_Spülmaschinen     1.20    8.26   3.34     18
Türart                        1.00    7.69   2.64     83
Lüftungsmotor                 1.85    5.78   3.84     78
Einschübe                     3.92    4.83   4.44     82
Ausstattung_Kombidämpfer      1.00    8.66   1.59     40
Reinigungssystem              1.68    7.15   4.25     48
Anschluß                      1.00    8.68   4.95     15
Kühlung                       1.00    7.35   2.24     43
Klimaklasse                   1.00    5.74   5.31     37
Kältemittel                   1.00    8.27   4.29     26
Pizza_Größe                   2.87    5.09   4.61     28
Beckengröße                   1.94    6.62   3.99     10
Anzahl_Kochstellen            1.00    5.36   6.55     25
Türanschlag                   1.00   10.37   3.27      9
Kippvorrichtung               1.00    9.67   5.28     22
Backkammer                    1.44    8.04   3.19     17
Pizzen_je_Backkammer          1.05    6.64   5.13     13
Grillfläche                   2.08    5.20   5.16      6
Bruttoinhalt                  1.15   10.62   5.54      5
Anzahl_Backkammern            1.00    6.08   6.64      4
Bautiefe                      2.00    5.35   5.68      3
Zündart                       1.00   11.96   5.64      4
-------------------------   ------   -----   ----   ----
Wgt. Average                  1.23    5.12   1.85      -

ℹ Over 90% of spans have lengths of 1 -- 2 (min=1, max=22). The most common span
lengths are: 1 (81.34%), 2 (13.78%). If you are using the n-gram suggester, note
that omitting infrequent n-gram lengths can greatly improve speed and memory
usage.
✔ Spans are distinct from the rest of the corpus
✔ Boundary tokens are distinct from the rest of the corpus
✔ Examples without ocurrences available for all labels

============================= Dependency Parsing =============================
ℹ Found 1528203 sentence(s) with an average length of 1.0 words.
ℹ 1 label(s) in train data
ℹ 1 label(s) in projectivized train data

================================== Summary ==================================
✔ 6 checks passed
⚠ 20 warnings

[shadeMe]

A couple of things to try:

• You could inspect the output of the tokenized text to see if there is some kind of misalignment between the tokenized spans and the ones in the training data.
• If you can confirm that the tokenization works as intended, you could try to train a spancat_singlelabel model with just those lables that have more than 100 examples to see if it improves its performance.

As such, the training data appears to have a lot of short texts with little context and similar-looking categories. Since that's further exacerbated by the low amount of data for many of them, the model is likely having trouble generalizing.