Availability Group is not connecting through Listener

I have set up 2 SQL VMs on Azure with VPN we connect to those VMs and it is a single subnet , No Public IP is used. but the listener is only accessible from Node1 when Node1's Availability group is primary and the listener is not accessible from Node1 when Node 2 Availability group is primary but Listener is accessible from Node2 when its AG is Primary.