Turning on GFCI breaker on subpanel trips main breaker

Check to make sure your spa heater is configured correctly for 240V operation

The Balboa VS501Z uses a solid-state control board that is configurable for 120V or 240V operation; however, when in 120V mode, it requires a hardwire jumper from neutral to a L2 (red) connection on the board to configure it, as per the manual. If that jumper is present on a VS501Z connected to a 240V supply, your breaker will trip immediately due to the jumper being a dead short from L2 to N, as you are observing. (If you're wondering why the breaker in the main panel trips but the breaker in the spa box doesn't trip, that's because what you have is basically a race condition between the two breakers, and whichever wins is determined by factors that are beyond our effective control.)