Does it make sense to discuss the "pressure" of the universe?

You are not taking it too literally. The only adjustment that you should make to your mental picture is that the galaxies are a gas with zero temperature and thus zero pressure. This is sometimes referred to as “dust.” As the universe expands, the density of galaxies decreases and the pressure remains at zero$^dagger$.

Note however that there are other components of the universe that do not behave this way. There is a small amount of radiation, which does have pressure. This pressure has been decreasing along with the energy density of the radiation as the universe has expanded. In fact, it decreases faster than the density of matter/galaxies. It used to be the dominant form of energy in the universe but is now very small due to this decrease. There is also dark energy whose density and pressure do not decrease, even though the universe is expanding. For this reason, it is now the dominant component of the universe.

---

$dagger$ Approximately. The galaxies and other matter in the universe do still have small velocities, but this is entirely negligible for discussing the behavior of the universe on cosmological scales. However, in the very early universe, before the matter had formed galaxies, there was a time when the matter was hot and had non-zero pressure. Then, indeed, as you say, as it expanded, it cooled and the pressured dropped until it had reached effectively zero. From that point on, we can think of the matter as pressureless dust.

An ideal gas in an otherwise empty universe doesn't actually press on anything, but it would press on an object that you hypothetically placed in it, and you can define the pressure by the effect the gas would hypothetically have on the object.

If you insert into the gas a thin barrier that is at rest in the local inertial frame of isotropy of the gas, then gas particles won't exert a net force on it but will bounce off of either side. The effect on the gas of two particles bouncing symmetrically off of either side is the same as if they'd simply passed through, and while individual bounces aren't usually symmetric, the time-averaged effect of all of them is. So you can also define the pressure of the gas as a momentum flux through an abstract surface in space. This notion of pressure is important, because it enters into the field equations of general relativity. (Technically, it's the individual particles that gravitate, but the pressure tells you their large-scale averaged effect on the gravitational field.)

Galaxies have nonzero velocities relative to the overall expansion, so they do pass through these surfaces, and their pressure is nonzero, but it's so small that it's usually ignored.

For a gas of photons (the CMBR), the pressure is large and in fact equal to one third of the energy density, with the factor of ⅓ coming from the three spatial components of the momentum.