Finding optimal threshold in multi-class classification task

Question

In a binary classification problem, it is easy to find the optimal threshold (F1) by setting different thresholds, evaluating them and picking the one with the highest F1. Similarly is there a proper way to find optimal thresholds for all the classes in a multi-class setting.

This will be a grid search problem if we do it brute force way. Any efficient way to do this?
Is there any package that ppl use or I can use for this?
Also is it common practice to find optimal threshold in multi-class settings, I couldn't any examples?

nimar · Answer

One solution is to explore a One-Vs-Rest classifier which creates separate binary classifiers for each class.

noe · Answer

In a binary classification setting, you normally only have a single probability, and therefore you need a threshold to define the decision rule.
However, in multiclass classification problems where labels are mutually exclusive, you have a multinomial probability distribution, that is, the $N$ probabilities of the input belonging to each of the $N$ classes, all adding up to 1. In this kind of scenario, the decision rule usually is simply picking the highest probability class.
In the cases where labels are not mutually exclusive, each label's threshold can be individually selected just like in the binary classification case.

Finding optimal threshold in multi-class classification task

2 Answers

Add your own answers!

Ask a Question