Using static vectors with text categorization model

[jamesmcaul]

Hello Spacy community,

I have been experimenting with using static word vectors to improve a text categorization model that I made. Since I was not using any word vectors before, I assumed that this should improve my model to some degree. However, after enabling these word vectors, I do not see any significant improvement (nothing above 0.001 increase in any p, r, f score for each of the categories I am training for).

My questions are...

• Is it plausible not to see any improvement when adding static word vectors to a model?
• If so, is my model somehow learning these word embeddings itself without the static vectors? Does this learning happen in the tok2vec component?

For context, I am using GloVe's word vectors. Here are the two config files for both the model without static vectors and with static vectors, respectively. Thank you!

config.txt
config_w_vectors.txt

[svlandeg]

Hi!

I'm not sure if it's useful to you, but this SO answer also deals with the difference between static & dynamic word vectors.

To answer your specific questions:

Is it plausible not to see any improvement when adding static word vectors to a model?
If so, is my model somehow learning these word embeddings itself without the static vectors? Does this learning happen in the tok2vec component?

Yes, I think that's entirely plausible, and yes, it would be the tok2vec component that is already learning good vectors. You can sort of "jump start" this learning process by providing static vectors, but it might not help much in accuracy at the end of the training cycle. You might notice a quicker uptake in performance though, at the start of the training?

[jamesmcaul]

Thank you! This makes sense. Yes I do notice a quicker uptake in performance during training when using the static vectors, but in each scenario the approach the same performance "limit". I assume that my tok2vec component is learning good enough vectors for the use case.