How does everyone manage a multi server environment and their credentials with ansible?

I’m trying to figure out the cleanest and/or most secure way to manage a multi server (linux) environment in AWX/Tower/Ansible.

I’ve seen a few posts around this topic, but there doesn’t seem to be a clear answer on which approach is the most secure.

Let’s say I want to use a single template/playbook against 20 different linux servers, all with different users and passwords / sudo passwords.

I can think of a few options;

A dedicated "ansible" user created on each host, with the AWX server or local server having the private key information. I think this would be fine in most cases, however if trying to perform operations needing root there’ll be issues with sudo passwords unless the "ansible" user has the same password on every server. It could be setup to have NOPASSWD sudo access, but this feels wrong.

Keep the current situation of having a different user and password combo for each host, but potentially specify the credentials in an encrypted (vault) vars file.

Have a separate template for each host each having the correct set of machine credentials for that host. This also feels slightly wrong and very cumbersome / hard to maintain.

Please let me whether I’m missing something obvious.