Source of magnetic dipole potential energy

One of the basic expression that goes without much thinking is the potential energy expression of a magnetic dipole in a magnetic field, $$ U = -mucdot B $$

In the case of electric and gravitational field, sources of potential energy, in general, can be realized by the conservative nature of the fields, meaning that these fields are curl free – you do work to change the position of the charge/mass inside the respective fields which are interpreted as a change of potential energy of that charge/mass.

But since magnetic field has non-zero curl, how can such magnetic potential energy be interpreted? And what is the source of that energy since magnetic field does no work? Though magnetic vector potential can be defined for divergence-less field, but it is not related to energy analogous to the general electric potential, is it?

As an example, suppose there is an uniform $B$ field along the length of an infinitely long solenoid. Keeping the solenoid’s current fixed, I bring a magnetic dipole from infinity inside this field. As a heuristic approach, I have several possible answer withouts explicit mechanism how they can arise the source of magnetic potential energy. I will just state them,

1. The agent does work.

2. The internal e.m.f. produced in the dipole does work (modelling the dipole as an amperian current loop) as the dipole goes through different flux region from zero $B$ at infinity to the uniform $B$ inside the solenoid. This form of explanation was given in section 8.3 of Griffiths as a source of energy when a dipole is attracted inside a non-uniform $B$ field. Is this explanation still valid for this particular case? But what if we extend the width of the solenoid to infinity to make $B$ as well as the flux in the amperial loop uniform in all space?

3. The current source who keeps the current fixed in the solenoid does the work.

4. Dipole steals energy from the $B$ field around it. As $B$ field contains an energy density of $frac{B^2}{2mu_0}$ in all space, the combined field energy inside the solenoid is less than the case when the dipole wasn’t there.

So, what is the exact source of magnetic potential energy of a dipole in a $B$ field?