Transfer switch connected to neutral and ground in sub-panel instead of main panel

Nope. This question comes up a lot on here, and you can't do it because it creates a paralleling problem with the neutrals.

These Reliance switches are designed for one thing: to provide a generator salesman a switch to offer to anyone without having to understand or investigate their panel. But they aren't well made, and they have a lot of limitations -- GFCI/AFCI support is a serious problem. And because of how they mishandle neutrals, supporting 2 separate panels is simply out of the question.

Your only Reliance-side option is to move the circuit in question so it's in the same panel with all the other circuits.

The better way to do it is to do what the generator salesman was trying to avoid: Evaluate your main panel for the feasibility of fitting a Generator Interlock. That lets you feed the entire main service panel (and subpanel too!) from the generator; on the proviso that you limit appliance use (or turn some off) to keep current draw within the generator's range. That will neatly moot your entire problem.