Why could the fumble in the Browns/Jets game not be advanced?

Two different rules could actually apply here, and neither mentions "forward" fumbles being advanceable anywhere in the current rulebook.

First is the rule for Fourth Down Fumbles, Rule 8.7.5:

If a fourth-down fumble occurs during a play from scrimmage:

(c) If the recovery or catch is by a teammate of the player who fumbled, the ball is dead, and the spot of the next snap is the spot of the fumble, or the spot of the recovery if the spot of the recovery is behind the spot of the fumble.

And then there is the rule for Fumble after the Two-Minute Warning:

If a fumble by either team occurs after the two-minute warning:

(c) If the recovery or catch is by a teammate of the player who fumbled, the ball is dead, and the spot of the next snap is the spot of the fumble, or the spot of the recovery if the spot of the recovery is behind the spot of the fumble.

Both would apply to this down (which was on 4th and 1, after the two-minute warning for the second half).

You are most likely confused due to the wording - as there are two different potential spots depending on if the fumble is recovered behind or forward from the spot of the fumble. Once Mayfield fumbled, there were two possibilities of where the ball might be spotted, unless Mayfield recovered his own fumble:

• The spot Mayfield fumbled (~1 yard behind the line to gain)
• The spot the ball was recovered, if that spot is behind the point Mayfield was at when he fumbled

No other result would have been possible, unless Mayfield recovered his own fumble. This has been the case for decades - since 1978, when the rule was introduced. This article by Football Zebras explains that in some detail. Note that the ball is dead immediately upon recovery by any other player than the original ball carrier - similar to a punt being touched by the punting team. Once Hunt recovered the ball (and had possession), the play was over, and nothing else could occur (for either team).

This is a 2 parter. First part will be the rule enforcing the call, second will be the reason behind the rule.

There is a special rule surrounding fumbles after the 2 minute warning of either half. Article 6. Fumble After Two-Minute Warning of the NFL rulebook states:

If a fumble by either team occurs after the two-minute warning:

a) The ball may be advanced by any opponent.

b) The player who fumbled is the only player of his team who is permitted to recover and advance the ball.

c) If the recovery or catch is by a teammate of the player who fumbled, the ball is dead, and the spot of the next snap is the spot of the fumble, or the spot of the recovery if the spot of the recovery is behind the spot of the fumble.

Note: The restrictions in (b) and (c) are applicable during a Try throughout the game.

So when Kareem Hunt recovered the ball, the ball was called dead since he was not the original fumbler.

[Second Part] Now we get to the more interesting question, why is there such a rule? The answer is the The Holly Roller, you can see the play here:

The 1978 Raiders fumbled their way to victory in a game trailing by six and 10 second left. Their QB Ken Stabler intentionally fumbled the ball right before getting sacked. You can see #40 also intentionally throwing the ball towards the endzone instead of taking the possession where it ends up getting scored by Raiders and giving them the win. Although the play was legal, it was clearly a loophole in the NFL rules, which was patched by the above rule in that immediate off season.