How can I account for an exhaustion effect when balancing a monster (e.g. in determining CR)?

Monsters are only roughly balanced

The Swarm of Rot Grubs is CR 1/2 and has a save or die ability. Harpies can enchant a creature and then kill it without provoking an additional save. Because a Monster's CR is only defined by damage and defence it does not capture the true threat posed by a number of monsters.

In addition all monsters can hit above their CR given the right circumstances. You will never be able to balance a monster to the point where a DM doesn't need to think about the encounter design. Causing exhaustion is just, as other answers note, particularly deadly because it can bypass hp which is a core balancing mechanic of the game.

If you want to use a monster that can cause exhaustion you can either make the monster weaker or balance the encounter in other ways. I will give some options for both.

---

Weaken the monster's ability

One way of accounting for the exhaustion effect when balancing a monster is to make the effect less debilitating in some way. If you do so then you will be able to rely more on the CR guidelines in the DMG. There are a number of ways to weaken an exhaustion effect. Here are a few options:

Cap the exhaustion level at 3: You mention this option in your question and it is a good idea. This will make Exhaustion less deadly and reduce the threat posed by multiple of these monsters killing a pc in one round.

Limit the effect to once per round: This can be done by having the player save on their turn (as with a Sword of Wounding). Since identical effects don't stack this might naturally limit the effect to once per round (per player) but if not you can also say so in the ability.

Offer additional saves: Have the player make an additional save at the end of their turn, removing all levels of exhaustion caused by the monster on a success. This dramatically reduces the chance of killing a player with this feature. If you have the initial save also remove all effects on a success it also reduces the threat of multiple monsters. Assuming no bonus to the save the chances of the exhaustion effect with a DC 10 killing a character are less than 1% assuming all attacks hit and the character isn't killed in the traditional way.

Immunity on a save: The ability is still dangerous (as one would hope from a monster) but if a successful save grants immunity to the effect there is less danger of reaching the deadly levels of exhaustion.

A Balancing Weakness: This would depend on benevolent DMing but if you give a dangerous monster a weakness, and let players plan to exploit it, you can make a deadly encounter significantly easier. This option will only work if you balance the encounter as discussed below.

---

Balance the encounter through other means

Avoid having a character get attacked many times

• Only have one monster with this ability
• Have the monster/s target different players (you might want to include a lore reason such as they are drawn to feed on the most energized targets)

Provide the players with an effective counter

• Something that will help with the saves or make them less likely to be hit.
• A convenient means to remove exhaustion (magic glade, magic item, spell scroll of Greater Restoration)
• Information ahead of the encounter to allow the players to prepare
• Terrain or other environmental factors that favor the players, especially features that make it easy for weakened players to escape

A DM can balance any monster for any level of players by controlling the monsters behavior and the environment. In the case of Exhaustion you will need to play the monsters and build the environment so that the effect doesn't kill your players or rather that they are only likely to get hit and fail their saves a couple of times.

This is hard to balance

Similar to the exhaustion-causing spell you linked, such a feature is hard to balance. The problem is that it stacks in a way that gets exponentially worse.

To clarify what I mean: if the encounter is against one of such monsters, the exhaustion-effect is mostly harmless, as the encounter is likely to end before the creature can apply 4+ levels of exhaustion against someone. If the encounter is against six of such monsters, the effect may kill a character that is way higher leveled than the monsters, depending on some luck on the initiative rolls and attacks/saving throws.

Monsters with similar consequences are the Shadows or the Intellect Devourer, which have effects that reduce an attribute and cause insta-death when the attribute gets to 0 (Str and Int, respectively) ¹. Both the mentioned monsters are topic of talk in forums due to their seemingly strange balance, with many people complaining they should have higher CRs. For example, How the hell are shadows CR 1/2?!

That said, from the CRs given to these monsters, it seems the devs considered the effect to be minor, and redirected the responsibility on balancing these monsters to the DMs using them (it is pretty much agreed that if you use these monsters as "just another monster of CR X", you are going to TPK your party).

Another monster that may be useful for comparison is the Night Hag, which has an effect that stops a creature from gaining the benefits of resting - effectively implying one level of exhaustion due to lack of rest, but also other problems (no HP recovery, no spell slots recovery, no feature recovery whatsoever). Again: the DMG does not clarify how the Nightmare Hunting feature changes the CR of the Night Hag, though, if at all.

So, in short: When designing the monster, it's likely that the feature will have a very minor impact, however, when designing the encounter, the DM will have to be extremely careful on how many monsters they are putting, at what situation (i.e., will the adventurers get any rest soon, or is this the first encounter and the party will be playing with a harsh disadvantage for the remaining of the day?), etc.

By the way, one major difference from the features I mentioned is that exhaustion only gets cleared one level at a time. However, there is precedent in the Exhaustion from Dashing (under the Chase rules in the DMG) to make it a "special exhaustion" that has all of its levels cleared in a rest, rather than just one.

So, to conclude: you will have to extensively playtest such a monster, as no guideline is given to anything similar, and theoretical balancing is considerably hard.

---

¹ The intellect devourer doesn't actually kill the creature just by getting the int to 0, but then it can use its Body Thief feature, effectively killing it.

Very Dangerous

The biggest problem with exhaustion as damage is that it's an exponential scale, but it also starts out quite strong.

1 level of exhaustion is pretty bad because it hampers your ability to escape a fight if there're ability checks between you and escaping (which there often are). A second level of exhaustion makes it nearly impossible to escape a fight. A third all but ensures you die in a fight, and anything past that is silly.

Consequently, monsters that do exhaustion damage are always no more than a few rounds from effectively killing a PC. Fighting these monsters essentially plays like fighting at levels 1-2 regardless of the party's actual level: combat is swingy, attacks that hit are almost always lethal.

When balancing these monsters, I've found it effective to treat their offensive capabilities as very high end in terms of damage, regardless of whatever actual damage or total CR they end up at. This means that if they have even just average defensive capabilities they will need some sort of penalty (like only being melee, moving really slow, having vulnerability to something, etc) to make up the difference.