Does food at room temperature spoil faster if refridgerated beforehand?

I'm not aware of it being true universally, but there are a few items for which it can be true.

One case is washed eggs (as sold in the US) -- if they were cold but you leave out and you're in a humid area, you'll end up with moisture on the eggs from condensation which can cause problems.

This might be true for other things that need to be kept dry. I'm not aware of this being an issue for milk or other liquids.

The short answer is no, that is a myth. Food spoils because of chemical breakdown and microorganism activity, both of these are slowed considerably by refrigeration. When you take food out of the refrigerator and it warms up these chemical and microbial processes speed up again, but they don't work faster because the food was previously refrigerated.

Some foods shouldn't be refrigerated, like bananas and avocados, but that's a different question.

As an addition to @GdD's answer.

Refridgeration slows down the mentioned processes, but it doesn't stop all of them, so the time in the fridge does count towards spoiling, though more slowly than time outside the fridge.

Assume some food lasts 10 days in the fridge or 1 day without refridgeration. It may be that it will last half a day out of the fridge if it spent its first 5 days in the fridge.
It may be that this is simply what is meant by your family wisdom: the food isn't as fresh after refridgeration as it was before.

In reality, the relationship between spoiling processes and time is not linear, and highly dependent on the process we're looking at. So the result could be anywhere between time in the fridge does not shorten shelflife out of the fridge at all and there's almost no time left until spoiled:

• extreme example: an unopened pack of food such as UHT milk or a tetra pak of tomatoes that could be stored at room temp almost indefinitely does not suffer in the fridge)
• simple chemical processes like fat going rancid are often slowed down to ≈ half speed by decreasing the temperature about 10 °C. So if you keep your butter at +8 °C in the fridge for a week, it will have gained approximately as much rancidity as if you'd kept it 3 1/2 days at 18 °C. And every day in the fridge will count roughly like 12 h out of the fridge.
• microbial spoiling is highly nonlinear with temperature.

I cannot think of any situation where refridgeration would prolong shelf life at room temperature after taking the food out of the fridge.

---

A different point is that food you put into the fridge (e.g. leftovers) is not immediately down at fridge temperature, so there will be a "history" of room temperature and cooling, which further decreases remaining shelflife.

Most fridges have a drier moisture content than the air in the room. They also transfer smells between items. A cool larder is therefore a good compromise: better moisture but slow-ish bacteria growth. Eggs for example do not need to be refrigerated is your kitchen/larder is cool enough. Your family's wisdom may come from observing foods being damaged in the fridge due to low moisture faster than if kept outside. A salad is an 'extreme' test case: it will shrivel in the fridge but rot outside...