Conservation of angular momentum and reference frames

You can understand this by looking at Newton’s second law first $$tag 1 F=frac{dP}{dt}$$ which works only in inertial reference systems. If we are dealing with a frame that is accelerating, then there are other non-Newtonian or pseudo-forces acting on the body.

Now if we look at your case and we have an object in a non-rotating inertial reference frame we would have that the torque is indeed $$tag 2 tau = frac{dL}{dt}$$ about the reference point.

Why is it necessary to assume inertial frames here?

However if we are in a reference frame where there are torques acting about other points that do not coincide with the centre of rotation, we would expect to find pseudo-torques or non-Newtonian type torques acting.

In such a case, equation (2) will not hold.

What would happen if the reference point was not fixed to an inertial frame?

Then, as explained, this would give rise to other fictitious torques which will complicate the original calculation. Additional terms would need to be added to equation (2) in a similar way as we would need to add additional terms to equation (1) when non-Newtonian forces are present.

A good example of this is the perturbation of planetary orbits by other massive objects. This lead us, for example, to the discovery of Neptune by noting perturbations in the orbit of Uranus.