AutoSSH through Firewall - Best practice for many devices on network?

My team is trying to find a solution to remotely access and update multiple Linux devices behind a firewall at several locations with their own firewalls. Traditionally, we would visit the business’ physical site and update the devices with Ansible running from within their network, or make changes via RDP and individual SSH access. Port forwarding is not an option, not even for a bastion.

||REMOTE SERVER|| ===> ||LOCATION 1 FIREWALL|| ===> [DEVICE 1, DEVICE 2, ..., DEVICE N]
(executing cmds)  ===> ||LOCATION 2 FIREWALL|| ===> [DEVICE 1, DEVICE 2, ..., DEVICE N]
                  ===> ||LOCATION 3 FIREWALL|| ===> [DEVICE 1, DEVICE 2, ..., DEVICE N]
                  ...
                  ===> ||LOCATION N FIREWALL|| ===> [DEVICE 1, DEVICE 2, ..., DEVICE N]

What is best practice with AutoSSH, or is there an alternative that would work better for maintaining these devices? We have two solutions so far.

Solution 1 – AutoSSH for every device:

-Establish AutoSSH connection from each and every individual device on each network (potential for hundreds per location). Configure SSH access for each of those devices.

Host loc_1_device_1
        HostName localhost
        Port 6000
        User maintenance
        IdentityFile /home/devops/.ssh/remote_maint

Host loc_1_device_2
        HostName localhost
        Port 6001
        User maintenance
        IdentityFile /home/devops/.ssh/remote_maint
Host loc_1_device_3
        HostName localhost
        Port 6002
        User maintenance
        IdentityFile /home/devops/.ssh/remote_maint

Host loc_2_device_1
        HostName localhost
        Port 7000
        User maintenance
        IdentityFile /home/devops/.ssh/remote_maint

Host loc_2_device_2
        HostName localhost
        Port 7001
        User maintenance
        IdentityFile /home/devops/.ssh/remote_maint

Solution 2-AutoSSH for proxy/jump host only:

• Establish AutoSSH connection from a single proxy (possibly a secondary as failover) on each of our client’s networks. Configure SSH access for each of those devices to a port on the remote server.

Host location_1
        HostName localhost
        Port 6000
        User maintenance
        IdentityFile /home/devops/.ssh/remote_maint

Host location_2
        HostName localhost
        Port 6001
        User maintenance
        IdentityFile /home/devops/.ssh/remote_maint

From there, connection seems easy enough with ssh -t location_1 ssh [email protected], or some ansible config for proxy command.

I’ve done a proof of concept for solution 2, running Ansible commands from the remote server, and it seems to work well, but there is pushback from other members on the team to go with solution 1 to remain a 1-1 inventory of every device (Which should just be the entire point of Ansible?). What concerns me with solution 1 is that there could be thousands of devices trying to maintain reverse SSH simultaneously, which could result in needless network overhead. It also seems like a needless amount of work for adding and removing devices from that network.

Is there another way to approach this? Am I on the right track, or am I crazy for thinking the rest of the team is crazy?

command="/home/tunnel/reverse_server/reverse_server.py",no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa <PUBLIC-KEY-FOR-ALL-DEVICES>