GCP internal load balancer firewall rules with GKE

I’m running a small kubernetes cluster on GKE, and am providing a VPN connection (CloudVPN) into this VPC. I would like to restrict traffic entering the VPC via the VPN to only accessing an internal load balancer (ILB) I have setup for this purpose (allowing access to a service running within GKE).
Unfortunately I’m struggling to work out the best way to restrict traffic for this particular use-case.
It doesn’t appear that I can set an ingress rule for traffic to the ILB, as GCP firewall ingress rules use tags to specify targets, which I can’t set on the ILB.

Any suggestions for ways to implement this functionality would be appreciated. Not set on using GCP firewall rules – but it seemed like this would be the obvious way to implement this. I’m currently using another instance as a ‘proxy’ of sorts (which I can firewall as I can set instance tags) – but having another instance running as well as the ILB seems like overkill. My next branch of investigation will be to see whether I can setup the CloudVPN in a separate VPC and then use peering and routes to implement something along this path – although I suspect there may be issues here, and would prefer a mechanism that acts on ip:port rather than just ip.