Does stronger regularization always improve performance on testing set?

Question

I am using the Sklearn logistic regression function to do a binary classification task on texts.
I did the task using three different inputs: Bag-Of-Words, TF-IDF, Doc2vec embeddings.
The question is that for the Bag-Of-Words, stronger regularization improves the performance on testing set.
However, for TF-IDF and Doc2vec embedding, stronger regularization doesn't improve the performance on testing set. Actually the performance downgrades with stronger regularization.
Why this is the case?

Ben Reiniger · Answer

Title question
The answer to the title question is a pretty clear-cut "no." For any fixed dataset, taking sufficiently strong regularization will make the linear model essentially constant, presumably not the best model.  Like most other things, there's a bias-variance tradeoff: increasing regularization weakens the model's ability to learn the training data, which to a point will tend to prevent overfitting and improve test performance.
Your more specific context
Word embeddings are substantially different from the other two, so the optimal regularization strength is probably mostly unrelated.
Between bow and tfidf, if nothing else the scale of the features will be larger in bow, so if you're not scaling, the penalties will be larger and the optimal regularization strength is likely to be smaller.

Does stronger regularization always improve performance on testing set?

One Answer

Add your own answers!

Ask a Question