Why do encamped Stormcloaks not fight when I attack a Thalmor patrol on a road nearby and carry the battle to them?

I met a Thalmor patrol escorting a Thalos worshipper at Location: 60164, -80436, Internal ID: 2358 (road between Helgen and Ivarstead) and ambushed them South of the Falkreath Stormcloak Camp (39501, -74990, Internal ID: 329). I tried to carry the fight into the camp for the Stormcloaks to be inadvertedly being attacked and therefore to join battle on my side. But they kept fleeing the Thalmor despite outnumbering them.
I wore Legion armour that time but have not joined the Legion.