A character at 10/26 HP is hit by a Wraith and fails the Constitution saving throw. They take 21 damage and their maximum HP is reduced to 5.
Are they unconscious because the 11 excess damage is less than 26 or dead because it’s greater than 5?
There is no general rule that I know of that would determine the order in which these effects would be applied in generally. However, the wording of the wraith's ability specifically tells us the order.
The wraith's life drain says:
...its hit point maximum is reduced by an amount equal to the damage taken.
Note that the effect says "damage taken" which means the character has already taken the damage. This means that damage must be taken first. This is an important distinction not only in cases like the one in your question, but in the case where the creature being hit with the attack has some form of damage reduction (see How does Vampiric Touch interact with Resistance/Vulnerability? for example).
So the damage is taken and then that damage amount is used to calculate the amount the character's HP maximum is reduced. That effect is then applied.
So, in this case, the character is unconscious with 0/5 HP.
For what it is worth, if the HP maximum reduction had come before the damage, the character would indeed be dead since the HP would go 26/26 -> 5/5 -> (-16) 0/51. This would be massive damage since the damage did more than twice the max hp of 5. This means the character would be killed instantly.
1 - Thanks @PL457
Correct answer by Rubiksmoose on December 8, 2020
Get help from others!