Which fact is true about the speed and frequency of light?

Then which color's speed are we referring when we say that light's speed in that medium (like glass) is c divided by refractive index? Is it average of all or what?

First, if the speed of light changes for different frequencies, then we say the refractive index is also changing as a function of frequency. Speed is always $c/n$, but $n$ can be a function of frequency.

Sources that give a single number for the refractive index of some material are just presenting a simplified view of the situation.

If you look around, you can often find experimental data on what that functional dependence is.

For example, for BK-7 glass, the $n(lambda)$, where $lambda$ is the vacuum wavelength, function looks like this:

You can see that the index only changes by about 1 or 2% over the visible wavelength band (~400-800 nm). We might easily just say $napprox 1.5$, if we don't need to know about the small variations due to frequency.

On the other hand, if we want to find the chromatic aberration of a lens made from this glass, then we'll need to know how much $n(lambda)$ varies before we can get started.