Debug registers (DRx) keep getting trampled when using with SetThreadContext

I’m trying to set hardware breakpoints from code by using Get/SetThreadContext and modifying DR0..3, DR7, but it only works if I do it repeatedly. Most of the time when I use Get after having used Set before, I see the values completely trampled (sometimes to 0s, sometimes to bogus numbers which are clearly not even addresses). Get’s return value is always 1. Sometimes it sticks for a while and the breakpoint hits. What could cause is and is there a way to track it? I tried setting a code breakpoint in SetThreadContext in ntdll, but it’s only being called from my code. It happens with and without VS attached. I must add it’s a game engine project, so it might be that some 3rd party lib does that.
The config is Ryzen 3900, Windows 10 20H2, Windows SDK 10.0.14393, VS 2017