What's the cause of sideways pressure (and why pressure is scalar)?

We can possibly explain it through symmetry.

Let's consider a cylinder partially filled with water and a piston that is used to compress the water.

In the absence of gravity all water molecules in the cylinder should be evenly distributed and there is not reason why the pressure - a result of their interactions - would be higher in one direction over other directions.

If we apply more force to the piston, the pressure in the cylinder will increase, but, still, the molecules will be evenly distributed (they would not now if they got squeezed by the piston or by the walls) and, if so, the pressure will still be the same in all directions.

If now we take an open cylinder with the gravity present, we can say that a tiny slice of water at the bottom of the cylinder is squeezed by the weight of the water above it and the distribution of molecules in that tiny cylinder should be roughly uniform - the same as if it was squeezed by a piston in the previous example. A tiny difference between the pressure at the top and at the bottom of the tiny cylinder will be too small in comparison with the pressure due to the weight of the water above it and, therefore, could be neglected.

Similar logic could be applied for a tiny cylinder in the middle or at the top: the pressure in any of them would be the same in all directions, being highest for the tiny cylinder at the bottom and lowest for the tiny cylinder at the top.

The idea that the pressure is the force for a given directed area suggests at first that pressure $P=F/A$, a ratio of two vectors, but this is not defined. Instead, say that $F=PA$, where $P$ can be defined, but is not uniquely determined by the two vectors as given.

Within the mode of your question - there are really two different pressure concepts.

One of them is exactly the (sorry for mentioning) stress tensor you said you did not understand, but actually you do - enough to see what is going on. The stress tensor is just the relation that gives the vector force on a vector directed area. Strictly, the pressure tensor is the negative of the stress tensor as pressure is the negative of tension (think of pulling a balloon out rather than pushing it in).

The other concept is the scalar pressure, which is the average normal force. The normal force is a scalar (just a number) - the length of the normal force on a unit area as a vector. If this scalar is taken in three orthogonal directions and averaged, this is what is called the pressure. It is also just a number. And since it is the average in all directions - it does not depend on direction.

If, for example, you get a block of rubber and push it down on the top but pull it out from the sides, then the force per unit area will be different for a horizontal surface inside than for a vertical surface. But the pressure will be the average. So, it is possible to have zero pressure even though the forces are not zero - if a small chunk of the material is being pulled and pushed so that the average is zero.

But, while you can do this to a fluid, such as water, the result would be that the water would flow out the side with the lower pressure. So, this could only occur for a short time. Actually, in this case, the inertia of the water starts to be important - that that introduces the idea of dynamic pressure, due to resistance of the fluid to motion. This introduces a number of complications including that the pressure can depend on the speed of the observer. So, there are quite a few conundrums in the pressure concept.