Comparing Multiclass classifiers with "No Answer"-Class

Precision, recall and F1 score are defined only for the binary case (2 classes), so if you want to apply that to the multiclass case, you need to apply a trick. A typical trick is to average the recall per class: Per class, you calculate which fraction of the words actually in that class are correctly classified. balanced_accuracy_score() in scikit-learn does that for you automatically.

Recall does not take into account false positives, so if you have words that where the model should say "no answer" (i.e. there are words where the ground truth is "no class"), then you should take that class into account. Otherwise it would benefit the model to just take a shot for all of the words it's given, and not classify any as "no class".