In vector algebra, why does the cross product of unit vectors like ${bf i}times{bf j}={bf k}$ become negative when the order is reversed?

This is a good question in a site dedicated to Physics since the reasons for the mathematical definition of a cross product in a three-dimensional vector space are deeply rooted in the needs of Physics.

Probably the very first case where the concept of cross-product emerges is with Statics. The study of rotational equilibrium conditions of solid bodies around an axis and levers' study motivated the introduction of a certain quantity. Such a quantity must embody the idea that the net effect of a force depends on its intensity, on the distance from the rotational axis, and on its projection on a direction orthogonal to the vector joining the rotational axis with the point where the force is applied. By a simple exercise of trigonometry, it easy to show that the intensity of such effective force (the entity we call torque) can be written as $| vec{r} | | vec{F} || sin(theta_{vec{r},vec{F}})|$. However, and this is the key point to motivate the definition of the cross product, such quantity must also have a direction (rotations may go in opposite directions, and opposite forces applied at the same point cancel their effect). Thus, to take care of the direction of the force, it is important to state that this effective quantity has a sign, and this sign coincides with the sign of the function $sin(theta_{vec{r},vec{F}})|$, provided that the angle is intended as the angle between the direction of $vec{r}$ and the direction of $vec{F}$, in this order, once a specific convention about what we call a positive rotation has been introduced. Notice that we have two possibilities, making a positive rotation corresponding clockwise or anticlockwise motion, and they are completely equivalent, provided they are used consistently. The usual convention assigns increasing angles to anticlockwise motion.

So, once we identify it as a useful quantity $$ | vec{r} | | vec{F} | sin(theta_{vec{r},vec{F}}) $$ (notice the absence of absolute value on the $sin$ function), and we have a convention about the sign of the angles, the order of the two vectors $vec{r}$ and $vec{F}$ maters for the sign of this expression. Indeed $$ theta_{vec{F},vec{r}} = 2 pi - theta_{vec{r},vec{F}}. $$ This implies that our quantity, let's indicate it by $M(vec{r},vec{F})$, must be anti-symmetric with respect to the exchange of $vec{r}$ and $vec{F}$: $M(vec{F},vec{r})= -M(vec{r},vec{F})$.

At this point, it is another trivial exercise of trigonometry to obtain an expression for $M(vec{r},vec{F})$ in term of the components of $vec{r}$ and $vec{F}$ in an orthogonal basis in 2 dimensions $$ M(vec{r},vec{F})= r_x F_y - r_y F_x. $$

Notice that in the case of 2D vectors, it looks almost like a scalar. Almost, because under inversion of the coordinates (but keeping the same force), it changes its sign. Since this is precisely the behavior of the oriented normal to the plane $x,y$, it would be attempting to introduce a third dimension and interpret $M$ as the only non zero component of a vector lying in the orthogonal ($z$) direction. Such identification is consistent with a more complicated, full 3D analysis, ending up in a natural way with the definition of a vector $vec{M}$ whose cartesian components are $ (r_y F_z - r_z F_y, r_z F_x - r_x F_z, r_x F_y - r_y F_x)$. Due to the bilinear character of the previous expression, we can introduce a cross product between 3D vectors such that $$ vec{M} = vec{r} times vec{F}. $$

All that is complementary to @Buraian 's reply to another related but different question . There, the anti-comutation reations ( $ {bf j} times {bf i} =-{bf i} times {bf j}$) was introduced by definition. Here, I have shown what are the physical reasons for that mathematical definition.

A final comment is that the story and motivations about the definition of the cross product are even much richer. I just mention that the possibility of embodying in a vector the properties we would like to assign to a quantity depending on two vectors is limited to a three-dimensional space. A deeper understanding of the cross product comes after understanding the natural way of dealing with antisymmetric multilinear products of vectors, which goes through the introduction of the tensor product and the so-called exterior algebra.

Nevertheless, I believe that understanding the simple physical motivations behind the cross product is a good starting point to appreciate much more formal mathematical concepts.