Multi-class classification with discrete output: Which loss function and activation to choose?

I’m working with a multi-class classification problem, using Keras Sequential models. In my dataset, the output class has one of the following values: (1, 2, 3, 4, 5).

Yes, I have already tried one-hot-encoding the output and using categorical_crossentropy as a loss function and softmax in the output layer.

However, I don’t think that softmax and categorical cross entropy is the right choice in my case. In my dataset, the output classes have a certain “discreteness” (or scale). Class 1 is the “worst” and class 5 is the “best”.

Let’s say if on a specific input the model predicts 2 and the true class is 1, it’s a much better prediction as when the model predicts class 5 and true class is 1.

I would like, that the loss function would take these “minor errors” into consideration and not treat all errors the same way.

To sum up, I’m wondering what my options are in terms of loss function and activation of the output layer for my given problem.

UPDATE: I have now tried to use mse as a loss function and replace my output layer with this:

model.add(Dense(1, activation='linear'))

The new method produces much better results, but I’m still wondering if there are any other options for my problem. Maybe a custom loss function?