Is it possible to join a modded Terraria game without downloading mods?

You always need the mod used to join a modded playthrough. There is no way to join without it, you will get kicked out for not having it by the game.