VMware + how to limit the balloon memory on linux machine

Ballooning only happens under some very specific conditions. For VMware ESXi to "inflate" the balloon driver in a VM, the host must be under very heavy memory contention, commonly less than 4% of free memory on the host.

Note that the balloon driver just asks the OS on the VM for memory, and any memory pages given to it are promptly deleted by the hypervisor since the VM wasn't using them for anything. So the balloon driver isn't actually using memory, it's just the way the hypervisor tricks the OS into telling it which memory pages it doesn't need.

If that doesn't solve the memory contention issue the host is having, the hypervisor inflates the balloon driver even more, to force the OS on the VM to start swapping memory to disk. This isn't ideal, but it's a lot better than the alternative, which is that the hypervisor start swapping to disk, since the OS on the VM actually knows what it's swapping to disk.

The way to limit ballooning is not the alter the presets as you've asked, but instead to assign a memory reservation to the VM that's more than 35% percent of the total memory of the VM (as the default limit for the balloon driver is 65%, reservations of 35% or less will have no effect on ballooning). This guaranties that the VM can get at least the reservation of memory, so this should in your case be the minimum amount of memory required to start your application on your VM.

Keep in mind that ballooning indicates very heavy memory contention of less than 4% free memory on the host, aka. that your host is almost completely out of memory, which is why it's taking such drastic measures.

Add more physical memory to the host, or move some VMs away from it to lessen memory demand!

Tweaking ballooning isn't going to fix anything! It's like applying a band-aid to a small cut to your hand while your leg is stuck under a car.