alternative to spherical harmonics to model radiance aproximation, optimized for real-time evaluation

I’m looking for a compact representation of radiance over the hemisphere that is also performant to evaluate in realtime (in a pixel shader). The purpose of it is to compress surface lightfields so that they are also very performant to render in real-time (in a VR application).

I found Efficient Irradiance Normal Mapping which is pretty neat, it proposes 6 cheap basis functions that produce the same error over the hemisphere as spherical harmonics up to l = 2.
Now, l = 2 may be fine for irradiance, but for radiance it’s too low frequency.
Looking at the results of Surface Light Field from Video Acquired in
Uncontrolled Settings, l = 4 spherical harmonics (25 coefficients) and l = 3 hemispherical harmonics (16 coefficients) seem to be enough to recreate some highlights and thus look promising.

What I’m looking now for is a basis that gives similar results like l = 4 SH or l = 3 HSH, but that is cheaper to evaluate at runtime.

I haven’t found any other research yet that goes into the direction I’m looking for, so I’m asking if someone has some good pointers?