How to configure X11 Forwarding so no one else on the same remote machine can display to my server?

Due to COVID my colleages and I work on the same remote Ubuntu machine in the office. We each respectively run VcXsrv on our local machines at home and SSH into the remote Ubuntu machine.

Occasionally, one of use will accidently configure out DISPLAY environment variables incorrectly and we start launching GUI applications on each other’s X server / machine.

Is there a way to configure the vcXsrv, or ssh/X11 forwarding such that we cannot display GUIs on each others machines?

FYI: we are all members of the SUDO group out of necessity…