Azure recently put Bring Your Own IP (BYOIP) into general availability. Great! We have been waiting for that for quite some time. We rolled up our sleeves only to learn that Azure services are not always compatible. We still managed to wrestle Azure and our specific requirements into a working setup, but as you will find out, it took some imagination and workarounds.

Our infrastructure is definitely not a cloud-first one. However, many challenges forced us to look for cloud deployments as a solution for our whole platform in a specific region. For one, slow hardware delivery time is getting even slower, with a chance of getting even worse.

Moreover, the Internet has been bordering off some markets, mimicking real-world borders. Since we are aiming for fast growth and expansion to new markets, we need new localized deployments to cover those segregated markets.

Azure seemed like a good workaround since it has many data centers spread around the globe.

Our infrastructure

To tell the truth, our infrastructure was not built with clouds in perspective. There are a lot of legacy requirements from a network infrastructure perspective. Like the requirement to support policy-based VPNs (Virtual Private Networks) with various vendors “on the other side”.

We were happy with Cisco ASA (Adaptive Security Appliance) solution for this since it is both a stable and compatible platform. We have tested Azure VPN (Virtual Private Network) managed solution, but when you try to configure VPN with someone who doesn’t have a lot of experience, hoping the “shut/no shut” will work “this time”, it can be frustrating not having a good enough debug, so you can assist them in what is they’re doing wrong.

Additionally, Azure VPN has limited capabilities – it does VPN well, but if you want to do some NAT on it, you better check if it can be done the way you need it. And while at it, check how many tunnels you can configure per gateway. We have locations with 700+ VPNs, so we would need to use VWAN to scale to that level in Azure.

And that is only a small part of the challenge. We figured we could simplify the design by encapsulating the known infrastructure design in Azure. On-prem, our services are visualized by VMware, and we have a talented team that handles it, not to mention our standard deployment process works best with VMware as well. The logical choice would be to simplify Azure deployment using Azure VMware Solution, also known as AVS (Azure VMWare Solution) (interesting abbreviation, right?).

AVS, however, does not support exposing Public IPs directly on VMs in AVS!
Checking options to merge the new Azure BYOIP feature “Custom IP Prefix” with AVS, we got a hard “NO, it is not compatible”.

It was back to the drawing board for us! It seems the Azure BYOIP feature is only compatible with “standard” Azure services like Azure Load Balancer. Or any other that specifically uses Public IP resource.

Plan F

Lucky for us, Cisco introduced a feature for a virtual ASAv security system in Azure. It now supports High Availability (HA) setup by integrating with Azure Load Balancer, meaning we can now attach our Public IPs. Fun fact, if you need a specific IP from your range, you can deploy it only via Azure CLI (Command Line Interface). To connect ASAv to AVS, we plan to use Azure VWAN since it will be more flexible in connecting other on-premises locations to it.

Additionally, VWAN enables us to do BGP peering between our Cisco ASAv NVA (Network Virtual Appliance) to simplify advertising the default route to AVS.
Oh, and it seems AVS “control plane” also likes to learn all specific routes (except the default one), and if you advertise more specific public routes via this BGP peering, you will break the AVS cluster! So, for safety’s sake, only advertise the default route.

Azure has limited capabilities in filtering where those routes advertised by ASA end up, so we had to create a separate VWAN instance from the one we use in other Azure locations. Otherwise, all our locations would start exiting towards Internet via this location where we advertised the default route.

And one last thing, if you copy our setup, make sure you find that small checkbox in the Azure VWAN hub peering towards AVS, where you enable sending the default route towards AVS!

Success? Well, for the most part

If you followed these steps, you should have a working setup with services deployed in VMware in Azure, exposed via your public IP, on a virtual Cisco ASAv device edge.

Is it perfect? No. This ASAv HA leaves a lot to be desired. Configuration is synchronized manually, and it is not developed with the BGP feature in mind, meaning standby ASA will also advertise routes even when it is standby. Exposing services on ASA behind Azure LoadBalancer is also limited in flexibility and is a whole new topic.

My best advice is that when you try to combine multiple Azure services, never assume they will be compatible. Always check inter-compatibility and limits. And try to set up PoC and at least test it without any load.

You can always raise a support ticket. Hopefully, MS support will get the convoluted way you imagined this to work and see whether they can support it (and how).