Back emf when circuit is closed and opened and the conservation of energy

Don't worry, electromagnetism is a bit of a handful at first. The best way, I found, to understand it is to always go back to Maxwell's equations.

"When a circuit is closed (by a switch), there will be a quick increase in current, which will induce a magnetic field in the solenoid. The same magnetic field causes a change in the magnetic flux linkage of the coils, which then produces a back emf. "

Assuming your circuit is a simple ideal solenoid and voltage source, the increase in current is linear and unbounded. This is an important point because it is the change in magnetic field that induces the back emf. Therefore your point:

"my guess is that the kinetic energy supplied to the electrons by the battery is the source of the back emf?"

is incorrect. It is the change of the magnetic field that is the source of the back emf, per Maxwell's equations.

Now, we're linearly increasing the current in the solenoid and you decide to open the circuit. What happens? The current wants to continue flowing and will force itself through the air gap to close the circuit.

"but this time the emf is in the direction of the battery. I don't understand how this doesn't violate the law. As I see it, there is no energy in the circuit to be "transformed" into the back emf."

The energy comes from the magnetic (and electric) field which also stores energy.