What's the equivalent of "what the hell are you doing?" in a negative sentence?

A very interesting observation. I hadn't realized that the hell was disallowed in negative contexts. Though, as starsplusplus has noted, in hell is not subject to that rule, and works fine. I'll have to add a note to my discussion of usage of the hell with a link to here.

As far as substitutes are concerned, there's a lot of research on Negative Polarity items like this; they're called Squatitives, from common expressions like

• They don't know squat (about it) = 'They know nothing (about it)'
• You can't do squat (about it) = 'You can do nothing (about it)'

and they have very peculiar syntax, as the data in the link above makes clear.

Since the pragmatic context for such a face challenge is emotional and uncooperative, many squatitives are either obscene or insulting. Definitely NSFW.

Keeping the word "hell" means that you could use:

Do what the hell you like!

You can emphasize the anything (or the lack of anything, as it turns out) with options like:

You can't do sh*t (to me / about it)!

You can't do squat (to me / about it)!

You can't do a damn thing (to me / about it)!

There's not one damn thing you can do (to me / about it)!

There's no way in hell for you to do that!

There is a glorious statement which many have used to great effect and satisfaction. The person at the receiving end will understand that you are impervious to the slings and arrows that may be shot in your direction.

Do whatever you want. I really don't give a flying fuck.

Yeah, well I don't give a flying fuck what you think! ^{Reservoir Dogs 1992}

I don't give a flying fuck if you're a cop. I don't give a flying fuck if you're God Almighty come for Judgment Day. Get out, or I'll twist your arms off your shoulders and beat you to bloody death with them.²

Cheap or whatever. I don't know. I don't give a flying fuck. Intimidation. He picked the wrong guy to try and intimidate. ³

The hell as a negation in the affirmative (an affirmative statement of negative sentiment, a non-negative negative.):

John Lawler covered most of the common uses of "the hell" and its variants in his linked article. I would add one other use that might be applicable to your examples.

"The hell" is sometimes used to negate a statement while at the same time providing emphasis or intensification. A statement may sound as if it is affirming a fact, but it is actually stating a negative. The hell I will means "I will not".

Note that there is no other form of negation used with "the hell" in this case. This differs from the examples that John Lawler and Hellion gave. (There is nothing wrong with their examples, mind you.) So "the hell" supplies the negation all by itself.

This use of "the hell" is actually short for "the hell if", as the following example shows:

Let's look, in this case, at your example. To say "You cannot do anything (to me)" might be a reaction to someone suggesting they can do something. Another way to put this might be something like The hell if you can do anything (to me).

Here you can see that the negating phrase is "the hell if" rather than just "the hell". You can substitute this more complete version in my earlier example, yielding "The hell if I will", where it works just as well. Leaving "if" out sometimes works and sometimes doesn't work.

A more complete verbal exchange along the same line as your example might be something like I can (verb) you for that!. An appropriate response might be The hell you can! meaning "You can't do that". Or The hell if you can do anything about it. meaning "You can't do anything about it."