What is it called when the majority moves and leaves behind a small group of people effectively exiling those people but without forcing them to move

There is no perfect term for both the movement and the act of leaving others behind.

For a start, try relocate and relocation.

We relocated to another table.

Our relocation left three people behind.

Merriam-Webster's entry for the verb "relocate":

transitive verb : to locate again : establish or lay out in a new place

intransitive verb : to move to a new location

Most dictionaries I consulted report "relocation" as the noun form with a corresponding meaning.

One caution: many words and phrases for this kind of movement have negative connotations due to the history of forced population movements or of white flight, the phenomenon of white families moving away from communities to avoid living next to black or immigrant families. Because the noun form does not signal whether people are relocating themselves or being relocated, rhetorical care is warranted. Consider the collocation "forced relocation," which refers to moving "large numbers of people under threat, planned and organized by governmental authorities, armed forces and/or militias." So you'll want to be careful with how you use and modify relocation.

It seems you've shunned the three you left behind—you have deliberately avoided their company.

Welcome to English Stackexchange, Carson.

Your question is similar, although not identical to this: I need one-word expression for "Ignoring someone intentionally"

The behaviour you are describing sounds like you are ostracising the person or people:

To ostracise, “To exclude (a person) from society or from a community, by not communicating with them or by refusing to acknowledge their presence; to refuse to talk to or associate with.

You could use migrate. The Cambridge Dictionary defines migrate as:

If people migrate, they travel in large numbers to a new place to live temporarily

or this might better suit:

to move from one place to another

Sounds like you’ve been repelled from your table:

to force something or someone to move away or stop attacking you

You might also consider that you’ve been displaced from your table:

to force something or someone out of its usual or normal position

Displace does carry political connotations, which I personally think adds flair, but to each his own.

In AmE: You and your friends ditched the others at the table. (Can also be used for sneaking out of a restaurant and leaving your date at the table.)