Is the velocity variable in the conservation of momentum equation always relative to the ground?

This law holds true for any inertial frame. Thus the velocity referred here, can be with respect to any inertial frame. The momentum conservation law states:

$$sum_{i=1}^N m_i u_i = sum_{i=1}^N m_i v_i$$

where, the subscript $i$ stands for $i$-th particle. $N$ is the total number of particles. $m_i$ is the mass of the $i$-th particle. $u_i$ and $v_i$ are the initial and final velocity of the $i$-th particle. As long as you are in an inertial frame with respect to these particles, the law holds true. Thus clearly, the velocities are not necessarily measured with respect to ground only.

EDIR: Please note that, as @zerothehero mentioned, all the velocities must be measured with respect to the same inertial frame.