Which side are these washers supposed to go?

It looks like the U-channel on that side came bent - If you compare it to the other side it seems to be bent quite a bit, judging from the pictures. This is why the caster plate isn't hitting the other side, and why you though to put the washers behind the plate. However, the washers are meant to go underneath the nut, not the plate. I would suggest trying to bend the channel back into shape - try using a hammer and a block of wood - place the block of wood on the side of the channel, and then whack the block of wood with the hammer. The block should help to minimize any distortion as you bend it back.

It looks like this product is poorly designed/built.

AS noted the washer goes between the nut and the caster plate.

Since the U channel is cheap and bending i would go to the hardware store and buy 16 larger washers (one for each bolt). They should be about a 1/4 of an inch, or more, wider that the U channel but have the appropriate size hole for the bolt.

Put the new larger washers between the caster plate and the U channel.

If the U channel still wants to splay outwards you can drill a hole through it sideways and install machine bolts, nuts, washers to hold it together, then tighten down your caster nuts.