Is there such a thing as too much training data?

[DarkSoliditi]

I working on an address model and have thousands of examples. I was going to label 1000 examples. Is that too much? Is there a general amount of test data you should have as a minimum?

[ljvmiranda921]

Hi @DarkSoliditi , there's no fixed rule in the amount of samples you're going to train. However, you still need to note the following:

• The size of your training set can affect your training time.
• Be careful with data imbalance re: your labels. You don't want your model to skew to one particular address
• Quality of your labeled samples. Are they clean and informative enough?

My suggestion is to try your 1000 examples first, check your data distribution (if they lean towards a particular label or not), apply the usual cross-validation techniques, and decide if you will still need more samples.

[DarkSoliditi]

Hey, Thanks for the reply. I'm very new to this and I'm not familiar with cross-validation. I am going to label 200 examples to validate against. My Dataset is good ultimately its 1200 unique and complete addresses so it should generalize really well in theory lol. could you link me some cross-validation techniques?

[ljvmiranda921]

No worries! My go-to reference is the scikit-learn cross-validation tutorial. They also provide utility functions so that it's easier for you to validate your model.

Note that if you'll use spaCy v3 via the config, you might want to generate separate .spacy files for your training and evaluation datasets.