How can I prevent cold air from dropping down my bath fan duct?

I am going to answer this even though i should probably wait for more information.

Bathroom, and dryer vent, outlet caps that exit a wall are generally attached to the exterior wall with screws, one at each corner. There may be some silicone caulking as well. They have a male crimped pipe that fits into the duct coming from the source.

You should be able to remove the screws and pull it directly out, turn 180 degrees and re-attach.

Here is YouTube video of one with a spring loaded flapper.

_{3" or 4" Round Duct Black Wall Cap w/ Damper, supplyhouse.com}

You can try to find an outlet with spring-loaded flaps. (Sounds like you already have one for the kitchen vent.)

Or, extend the outlet with a 90-degree elbow. That way the end will be facing sideways, not down.

You could replace the soffit vent with one that has a flap to prevent air coming inside, maybe something like that. But if it's on the second floor, you'll need a tall enough ladder.

Or you could pull the fan out from the bathroom (or visit the attic) and check if it has a one-way flap. Maybe it has one that is jammed open by cobwebs or something, and all that's necessary is a bit of cleaning. If it doesn't have a flap, then you can add one between the fan and the exhaust pipe. I'm not sure how it's called in English, but it looks like that:

Make sure you pick a model that works in the orientation it'll be installed in.