What exactly is moment? Why does it correspond to a rotation?

I will ask one question: what about the moment of inertia?

I think understanding why such a way of defining moments is useful is much more important than investigating the definition of a moment itself.

If you have a solid example of a certain concept, it's much easier to extend that to a more general case and find a comprehensive reasoning. So I will take the moment of inertia as an example, as well as the moment of force (torque).

The moment of inertia is defined as: $$I = int dm r^2$$ where $r$ is the distance from the rotational axis and $dm$ is a mass infinitesimal or, mathematically, the integrating variable.

So, the first thing you may notice is that $I$ is a scalar quantity. The moment does not have to be a vector quantity. That's one thing. As the Wikipedia page states, the moment can be either vector or scalar, and we can call anything in the form $r^ntimes appropriate physical quantity$. Perhaps you can think of $r$ as being the magnitude of the vector $vec{r}$ so that we can see the identicality of the scalar moment and the vector moment.

I will now move onto your second question: what determines the direction of the moment vector, for example the torque (the moment of force)? As you can see from its formula $vec{tau}=vec{r} times vec{F}$, the operation that relates the radius (moment arm) vector and the force vector is cross product. If the torque were to be defined as $tau = vec{r} cdot vec{F}$ using the dot product (which doesn't make any physical sense at all) then the torque would have been a scalar, not a vector. So the fact that the torque vector is orthogonal to both the radius vector and the force vector precisely comes from the pure mathematical properties of the vector cross product operation. There's nothing to do with the physics here, other than that we can think the direction of the torque vector as being the direction of the rotational axis.

I will finish by leaving another comment on how we determine the positive/negative directions of torque vectors, which is namely the right hand rule. The right hand rule is followed not because it is a mathematical truth, but because mathematicians and physicists all over the world just chose to set that as a standard convention.