Is GTA Online protected by an anti-cheat engine?

The servers are, yes.

Hackers/Cheaters get banned every ban wave - which seem to be every couple months (unless a serious hack/cheat is found and a hotfix is pushed).

Definitions of hacks/cheats are cloudy, but this article suggests it's 2 things

We’re trying to find out exactly what the ban criteria is, but it seems to be anyone who’s used either a cheat engine or a DLL injector – things like Script Hook and the Field of View mod.

If a hacker/cheater drops you money, I'd be cautious to pick it up, because as the same article says

There’s also some uncomfirmed reports that if you happened to have received $GTA from a modder then you have been banned too, but then some people who did this have been banned, others haven’t

This is an e-mail from Rockstar detailing what will get you banned;
_{(source: gta5cheats.com)}

You can never really stop hacks/cheats from being used. This steamcommunity posts explains some things about the ban, but doesn't cite sources, so I'm unsure if they're reliable.