Loss function in order to find the most certain samples

I have a classification problem where I try to predict if some measurement will go up or down in the next 10 seconds.

I have a few hundred of features and so far I simply trained a few classification models and measured their ability to seperate the two population.

The issue is that enentually I need to return only a few dozens of samples among the thousands of samples in the test set that I’m highly certain about them, let’s say the top 15 samples that will go up and the 15 samples that will go down.

For this I’m using two thresholds and I’m returning only samples that the score of the trained model passed the upper threshold or if it was lower than the lower threshold.

My question is if there a more suitable loss function to train the model for this case (currently I’m using logloss and auc) where I dont really care about most of the population and I only try to return the samples that I’m the most certain about them.

Thanks