For the wraith, the damage comes first then HP reduction

There is no general rule that I know of that would determine the order in which these effects would be applied in generally. However, the wording of the wraith's ability specifically tells us the order.

The wraith's life drain says:

...its hit point maximum is reduced by an amount equal to the damage taken.

Note that the effect says "damage taken" which means the character has already taken the damage. This means that damage must be taken first. This is an important distinction not only in cases like the one in your question, but in the case where the creature being hit with the attack has some form of damage reduction (see How does Vampiric Touch interact with Resistance/Vulnerability? for example).

So the damage is taken and then that damage amount is used to calculate the amount the character's HP maximum is reduced. That effect is then applied.

So, in this case, the character is unconscious with 0/5 HP.

---

For what it is worth, if the HP maximum reduction had come before the damage, the character would indeed be dead since the HP would go 26/26 -> 5/5 -> (-16) 0/5¹. This would be massive damage since the damage did more than twice the max hp of 5. This means the character would be killed instantly.

^{1 - Thanks @PL457}