if computer is compromised, does hardware key still protect in ssh?

When I am 'away' can the threat actor access the server (assuming I logged out)?

This will largely depend on the setup of your hardware key, assuming it is left inserted. If your key is set to require physical touch every time it is used for authentication, this likely cannot be bypassed, even if your system is compromised. If the device requires a PIN, the attacker likely needs to steal that first. In the case of both features, the device may not require them again for a period of time, at which point you'd likely be exposed.

when I am 'sitting there using it' can the attacker move into the server and plant access directly to the server so that they no longer are impeded at all by the key.

If the attacker has root, they can typically manipulate all files and processes on the system. It is not infeasible that they could backdoor the ssh binary to run extra commands, or attach to it at runtime to inject extra stuff.

It would seem the hardware key is most effective when it is unplugged or requires additional, unforgable physical interaction in order to activate each time a sensitive operation is performed. However, as alluded to above, while a hardware key is a powerful tool as part of a defense in depth strategy, it does not remove the need to focus on securing the rest of the system.