Xorg has high CPU load after unlocking XFCE

I am running Xubuntu 20.04 with a NVIDIA GeForce GTX 760 and an AMD Ryzen 7 CPU.

The Nvidia driver version is 450.66, the kernel is 5.4.0-47-generic

Starting up without previous hibernation works fine, no excessive CPU load. Invoking hibernation works fine, no error entries whatsoever, resuming is fast and without any problems (checked dmesg and journalctl).
After logging in from lightdm top shows a CPU load of 100% of Xorg. The high CPU load persists for up to 5 Minutes, on one core mostly.

I’ve read some similar threads (e.g. the XModmap bug), checked the xfce4 config files for empty values, but found nothing.
So any idea where to look would be appreciated.

Edit
Seems to be a bug in XFCE