How to automate the key exchange in WireGuard when you deploy a cluster of machines?

The githubixx wireguard ansible role might be what you are looking for. The author of the role says:

I use WireGuard to setup a fully meshed VPN (every host can directly connect to every other host) and run my Kubernetes (K8s) cluster at Hetzner Cloud (but you should be able to use any hoster you want). So the important components like the K8s controller and worker nodes (which includes the pods) only communicate via encrypted WireGuard VPN. Also (as already mentioned) I've two clients. Both have kubectl installed and are able to talk to the internal Kubernetes API server by using WireGuard VPN.

I reviewed the tasks and the template, but I haven't tried it out yet myself. It supports different versions of linux. It installs wireguard. It handles key generation, distribution, and wireguard configuration for a full mesh VPN. For each host it configures an interface with the host's private key as well as peer configuration for all the other hosts using their own public keys. You can set variables in the ansible inventory for each host (e.g. addresses) to tailor the configurations to your network.

I've got an Ansible role that may help you. It was designed to act as a gateway, but I think it may still work for you (hopefully without having to tweak it).

The most helpful aspect is that it can run in either client or server mode. So what you would do in your playbook:

1. Run it on master in server mode,
2. Save the generated public key to a variable.
3. Loop though the workers running it in client mode providing the server's public key.
4. Collect all of the public keys into a list variable.
5. Rerun it on master in server mode, this time with all of the worker keys.

Then, you should be good to go. I normally don't put all of this together, just do a single run for each device, but if you want to automate the entire thing (especially if you have many workers), then you should be able to get all of that into a single playbook.

For more information, see the role documentation.

I don't recommend sharing keys across devices for the reason you mentioned.

(Disclaimer: As the question has not received an answer yet, I came to the conclusion that there is currently no out-of-the-box solution. Thus, I'm sharing what I ended up doing, and hope it will help others. Still, I would be interested to learn of a better solution if there is one.)

---

WireGuard can be used, but each of the n-to-m connections will need its own keys. Managing that by hand becomes tedious if the number of machines increases. Therefore, it is advisable to automated it:

1. Generate all keys in advance
2. Distribute the keys to the machine using Ansible (e.g. by generating systemd services wg0.network and wg0.netdev)

Note that step 1 assumes that the maximum number of machines is known in advance. If you cannot provide an upper bound, this approach will not work. Instead you will need to implement a dynamic key generation and exchange protocol.

If you end up using fewer machines then the configure upper bound, it will still work. Distributing the keys means that WireGuard will allow connections from different machines, but does not require that all peers are always available.

---

Scaling: In my setup, the upper bound was in the order of 100. That works without problems, but I did not test how it scales if you need to go higher (e.g. to the order of 1000 or 10,000).

What if there is no upper bound? -- That is a more difficult problem, and I don't know how to solve it best. As said, you would need to generate key dynamically and exchange them. I was considering opening SSH and using it to exchange on-the-fly generated WireGuard keys.

Note that with SSH, you can reuse the same key on different machines. That eliminates the need to exchange dynamically generated keys. But be aware that reusing the same key for all machines is a trade-off between convenience and security. If one machine gets compromised, the attacker will gain access to all other machines, too.

Maybe there is some solution that is more elegant and more secure. Perhaps directly using Wireguard is not a good idea if you need to support an arbitrary number of dynamic hosts. It is easy to introduce subtle security issues when implementing your own key exchange protocol.