Clients Can't Connect to Modded Minecraft Server with Optifine

I’ve started a 1.16.1 Minecraft modded server using Apex Hosting a short while ago, and frequently encounter an issue when it comes to Optifine.

If a client with Optifine installed tries to connect to the server, they will get an error about the client’s mod list not matching the server’s mod list.
Adding Optifine to the server fixes this issue, allowing them to connect, BUT it’s not a mod designed to be had on the server. It spits out frequent errors which will be slowing down the server’s performance.
We also tried the server without Optifine and the client not having Optifine, and that allowed them to connect as well.

Any idea why it seems like we need to have Optifine installed on the server?

Can confirm it is just Optifine that we are changing, here. We’re not adding/removing other mods from the client/server when trying to get this to work.

We are using Optifine version 1.16.1 HD U G2, in case it is relevant. Might also be worth noting it’s a custom collection of mods, and not a modpack.

Any help will be greatly appreciated!