Bad results for training set and good results for test set

Question

When training an algorithm, I have ROC = 0.5896, Sensitivity = 0.3333, and Specificity = 0.8375. When considering the test set, Sensitivity = 1 and Specificity = 1. This could happen or is a problem?
Another question:
When comparing several learning algorithms, I choose the best based on the results of the test set?
Any help will be appreciated.

Michael Solotky · Answer

Seems quite strange. I bet you've got a bug or test set size is small enough. Like if there're only 2 objects: 1 positive and 1 negative, you could have classified them correctly even with a random classifier. I assume your test set is not that small, which means there's a bug somewhere. Also, in theory, if both of your sets are large enough and are drawn from the same data distribution, you can usually have one of 2 types of behavior:

about the same result both on training and testing sets (when your model underfits or fits well)
better results on the training set then on the testing (when your model if overfit)

Answered by Michael Solotky on October 16, 2020

about the same result both on training and testing sets (when your model underfits or fits well)
better results on the training set then on the testing (when your model if overfit)

Bad results for training set and good results for test set

One Answer

Add your own answers!

Ask a Question