How to measure a almost (?) ordinal classification?

Question

I have a model where I predict classes to define instructions for a trader robot. The classes are -2, -1, 1 and 2 (strong sell, light sell, light buy and strong buy) and I'm using a simple confusion matrix to asses the performance of the model, however I would like to know if there are other ways that takes into account the "distance" between the classes.
I've found some content about ordinal regression, but I already made my classification in other way (not sure if this content applies to me though).
Any suggestions?

aivanov · Accepted Answer

You could just calculate the mean absolute error.
However, this quality measure (and any other measure taking into account the distance) would not be consistent with the loss function you used while training your classifier I guess. So it would not be fair to your algorithm. It is like if your professor would change the grading system after you submitted your test.
Instead of changing the quality measure you might want to reconsider your approach. The suggestion to use the ordinal regression sounds good. In case if the cost of misestimation between neighbors is similar you could even use plain regression and then round.

How to measure a almost (?) ordinal classification?

One Answer

Add your own answers!

Ask a Question