Can you explain the direction of angular displacement?

This question basically boils down to the question how do I represent 3D rotations in the most natural way?. You might know that you can describe rotations using matrices but this is not the most compact way (although it is still a useful representation especially for doing calculations). The minimum information you need to describe a rotation in 3D is the axis about which you perform the rotation and also the angle.

You can represent an axis using a vector: the direction of the vector gives the direction of the axis. This is why the rotation vector (read: angular displacement) points in such a weird direction. It doesn't give you an actual displacement, the direction of the vector is just to indicate the axis of rotation.

Once you have the axis of rotation you still need to specify the angle. We didn't specify the length of the vector so we define the length of the vector to be the angle of rotation.

There is only one problem in our definition. When you use a vector to specify the axis you have two options: you can point the vector in either direction of the axis. The right-hand-rule is there to ensure we always pick the same option. We could also have defined a left-hand-rule as long as we always stayed consistent.