How to choose a good operation point from precision recall curves?

Question

Is there any standard method to determine an "optimal" operation point on a precision recall curve? (i.e., determining the point on the curve that offers a good trade-off between precision and recall)

Thanks

SheldonCooper · Accepted Answer

The definition of "optimal" will of course depend on your specific goals, but here are a few relatively "standard" methods:

Equal error rate (EER) point: the point where precision equals recall. This feels to some people like a "natural" operating point.
A refined and more principled version of the above is to specify cost of the different kind of errors and optimize that cost. Say misclassifying an item (an error in precision) is twice as expensive as missing an item completely (error in recall). Then the best operating point is that where (1 - recall) = 2*(1 - precision).
In some problems people have a natural minimal acceptable rate of either precision or recall. Say you know that if more than 20% of retrieved data is incorrect, the users will stop using your application. Then it is natural to set precision to 80% (or a bit lower) and accept whatever recall you have at that point.

Pritesh Gohil · Answer

Yes, the point closest to (1,1) is one way to choose an optimal threshold. This can be done systematically with recent publication such as LRP error or F1-score based method to select the optimal threshold.

In the first method, the optimal confidence threshold is at which minimum LRP error is observed LRP Metric.
In the second method, the optimal confidence threshold is at which maximum F1 score is observed F1-score based method.

Hadley Is My Hero · Answer

I'm not sure how "standard" this is, but one way would be to choose the point that is closest to (1, 1) -- i.e. 100% recall and 100% precision. That would be the optimal balance between the two measures. This is assuming you don't value precision over recall or vice-versa.

Answered by Hadley Is My Hero on January 5, 2021

conjugateprior · Answer

Following up on SheldonCooper's second and third bullet points: The ideal choice is to have somebody else make the choice, either in the form of a threshold (point 3) or a cost benefit tradeoff (point 2). And perhaps the nicest way to offer them the choice is with an ROC curve.

Answered by conjugateprior on January 5, 2021

How to choose a good operation point from precision recall curves?

4 Answers

Add your own answers!

Ask a Question