How did Landa think his plan would work?

Because it happens all the time. People on the side of "good" tend to be willing to let bad guys get away with things if it means they change sides. The police frequently release small time criminals in exchange for information on bigger fish. Likewise, when Landa allowed Hitler and the rest of German command to be blown to bits without interference, he sealed the deal that he got to live.

One of the best reasons to go through with deals like the one he struck, from the perspective of Allied command, is that if you become known for treachery against defectors, who in their right mind would defect? However, if you honor deals with scum when they have a good enough offer, you gain a reputation for being worthwhile to defect to. Essentially, Landa was depending on the good guys to be honorable and follow through with their agreement.

What he failed to understand is that while Aldo is on the side of the good guys he is not a good guy. The entire reason he was in Germany was to be a nightmare for the Nazis. He'll follow through with just enough of the deal to make his superiors content, but in truth, he just wants to kill some Nazis. So when it's time to honor the deal? Nah. That just doesn't sit right with him. And the worst that he's going to see anyway is a bit of being chewed out... And he's "been chewed out before..."

It did work.

The loss of his radioman and his new scar notwithstanding, Landa's plan was implied to work. As Aldo is preparing to mark Landa, he remarks what a great deal Landa's proposal was for the Allies, prompts Utivich to agree with him, and concurs that he in fact would take (and presumably uphold) the deal if he himself were in charge.

ALDO: Well, if you're willing to barbecue the whole High Command, I suppose that's worth certain considerations.

The main reason he marks Landa is because he expects him to actually get what was promised and it doesn't sit well with him that Landa would get off scott free for all his previous crimes.

This is also reinforced by Aldo's line when he shoots the radioman:

LANDA: I made a deal with your General for that man's life!

ALDO: Yeah, they made that deal, but they don't give a fuck about him. It's you they want.

It is clear that Aldo expects the Allies to honor their deal with Landa. He even expects to get chewed out for shooting the radioman and carving a swastika into Landa's head. Landa's miscalculation was not in trusting the Allied command, it was in his faith that Aldo and his men would remain strictly obedient to the Allied command in delivering him and his radioman safely; after all, he heard the general order Aldo to deliver Landa and the radioman to them safely. And even with this miscalculation, it appears at the end of the movie that he will get his deal.

The answer lies in the very reason when he is betraying Hitler. He can see that it's a lost cause and sooner or later Germans are going to loose the war and officers like him would be hunted down.

Smart as he's set up to be, he's making deal with US Army's high command well in advance for a peaceful life in return of saving millions of lives, time, effort, money.

He's putting his trust on US Army's words as he has no option because that cinema was one of the rare chances when top leads of German High Command were in one place.