What causes peculiar velocities of galaxies?

Velocities of galaxies

When galaxies are gravitationally bound to each other in groups or clusters, they move on more or less elliptical orbits in the common gravitational potential from all the other galaxies (as well as all the dilute intracluster gas which is also a significant part of the total mass). I say "more or less" because galaxies do occasionally come close enough to each other that the potential is dominated by individual galaxies. That is, although some may move straight toward the center ("collapse") and some may rotate around it, in general the orbits will be something in between.

This is completely analogous to stars in a globular cluster, which also don't collapse or move in pure circular orbits.

The velocity distribution of gravitationally bound galaxies thus depends on the total mass, and is indeed observed to have dispersions $sigma_V$ of the order of a few 100 km/s for small groups (e.g Carlberg et al. 2000), to 1–2000 km/s for massive clusters (e.g Girardi et al. 1993; Karachentsev et al. 2006).

Decreasing peculiar velocities

You're right that a particle moving with a non-zero peculiar velocity in an expanding space asymptotically will come to a rest (if you're careful to choose the right definition of "joining the Hubble flow", see Barnes et al. 2006). However, groups and clusters have "detached" from the Hubble flow, being gravitationally bound, so they will tend to keep their velocity distribution (I say "tend to", because galactic encounters/merging eventually may cause galaxies to lose energy and sink to the bottom of the potential well on extremely long timescales).

Velocities are larger in clusters than outside

As I said, the velocity distribution depends on the mass $M$ of the cluster. In fact, for a cluster of radius $R$, $$ sigma_V^2 = frac{GM}{cR}, $$ where $G$ is the gravitational constant, and $c$ is a factor of a few which depends on the exact geometry and mass distribution of the cluster (e.g. Binney & Tremaine 2008).

So you're right that galaxies which are not part of a cluster — so-called field galaxies — have smaller velocities than those in massive clusters. However, the peculiar velocity of a lonely field galaxy is not easily observed, because — in contrast to cluster galaxies — you cannot compare its velocity to anything. Velocities are measured from the galaxies' redshifts, and in a cluster you can take their average velocity as the "systemic" velocity, i.e. the velocity of the center of mass. But for a field galaxy, you don't know how much of its redshift is cosmological, and how much is due to its peculiar velocity. If a galaxy contains a standard candle — e.g. a Cepheid or a supernova — so that its distance can be measured, it is possible to obtain its peculiar velocities. Measured values are typically a little smaller, but not much, that the velocity dispersion of group galaxies, usually below a few 100 km/s (Tsvetkov; Wesson 2005).