How to remove the guardian temple's mining fatigue animation?

Without a little work, this can't be done. You can, however, create a resource pack where the sound file and image for the effect is blank. You will still get mining fatigue, but you won't be scared.

I originally thought that this could be done with a resource pack and was going to write up a quick tutorial explaining which files to edit, but it turns out there isn't a good solution to this.

There's no special texture for the elder guardian ghostly effect. It's actually a transparent version of the elder guardian texture, and uses the same model:

So, even if you did know the exact file to change, you'd be changing the texture of the actual elder guardian, which I would guess is not something you want to do.

Technically, this is still possible by changing the elder guardian's texture, then having all elder guardians in your world by invisible, and instead use an armorstand with an item that has the elder guardian model (And constantly teleport it to the invisible elder guardian). However, this would mean the swimming and spikes would not longer be animated (logically).

So to conclude, there's no good way of doing this.

I know a way: turn off screen animations then turn off hostile mob sounds. You will still see the image but not the sound, I understand. It makes me jump every time.