How can I get my floor register to fit in the duct opening?

You can try one of these:

1. Move the register opening.
2. Change the size of the opening so a new register will fit.
3. Order a custom sized register that will fit your new opening. It won’t match the design of your existing registers, but it’s an easy fix for your new duct opening.

That's a custom cut register to fit so close to the wall; you don't want a new one or you'll just have to do that again (which isn't fun).

The problem is the mangled boot. Straighten it as best you can with some pliers. Then attach it like it's supposed to be with some screws into the flooring/subfloor. Find the corner it likes to sit in, biased to either side. Screw the side in with two screws. Then the hard part: screw it down to the room side (may need a right angle drive, and pilot holes). Or screw it down to the wall side after you slip some wood back there to preserve the clearance from the wall.

When you screw it in make it as flush with the floor as possible. When you go to put the register back, stick it in at a 45 and then slide it sideways as you push it down. That may be all you have to do but the problem will persist, and it won't be as air-tight as it could be. Ideally the boot is taped to the floor which the register will hide.

Measure the opening in the metal duct and buy a register that will fit that opening. There are metal ducts that are 2 1/4" high and 3 1/4" high with widths at 10", 12", and 14", Also, turn the white register over and measure the size of the protruding piece holding the moveable vanes. If that dimension is larger than the metal duct opening then that register will not work. Remember, the protruding piece of the register that holds the vanes has to be able to fit into the opening in that duct. Hope this helps

If i understand correctly the register fit before you installed new baseboard trim and now it the trim is causing a miss-alignment. (My answer is based on that assumption, let me know if that is not correct.)

I would suggest that your best option is to remove the baseboard, put the register back in place then hold the baseboard in place on top of the register and mark on it where the ends of the register are. You will need to trim (cut) a notch in the bottom of the baseboard so when it is on the wall the register will be underneath it (recessed in the notch).

If you cut the notch too shallow/short you will not be able to remove the resister so do some testing before you attach the baseboard to the wall.