View jumping in VR games on Xperia Z3 compact

After the update to Android 6.0 the steering/looking around in my VR games seems to be a bit buggy.

I’m using a Sony Xperia compact Z3 and the problem is the following. In apps (such as Google Cardboard) I can look around, but after some random time the view will center at a certain place. This makes looking around nearly impossible.

Does anybody have any idea what the problem here is and how to solve it?