Azure Virtual WAN is one of the networking options offered by Azure, and routing inside of Virtual WAN has become a sort of dark art with the many supported options. Some years ago Routing Intent (also known as Routing Policies) was introduced to simplify routing, but at times you need to go back to the “old” way of configuring routing.
Many posts have been written about routing in Virtual WAN, and this week I found myself looking again into the “indirect spoke” design, also known as “Routing through an NVA” in Azure docs, and how a recent feature called “Propagate static …
Azure Virtual WAN is one of the networking options offered by Azure, and routing inside of Virtual WAN has become a sort of dark art with the many supported options. Some years ago Routing Intent (also known as Routing Policies) was introduced to simplify routing, but at times you need to go back to the “old” way of configuring routing.
Many posts have been written about routing in Virtual WAN, and this week I found myself looking again into the “indirect spoke” design, also known as “Routing through an NVA” in Azure docs, and how a recent feature called “Propagate static routes” can dramatically simplify the configuration.
The design
The following diagram illustrates the challenge at hand: workload VNets are not connected directly to Virtual WAN, but instead they are peered to “user hubs” that contain network virtual appliances or NVAs. If you want to read a philosophical rant on why this might still make sense from an architectural perspective, feel free to have a look at my post Is the core/distribution/access design dead? Enough digressing, coming back to Virtual WAN now this is what I am talking about. Here you have a sample topology with two virtual hubs in the same virtual WAN, both with “indirect spokes” attached to them:
You can check the effective routes for the virtual hubs either in the Azure portal, with PowerShell or in the Azure CLI. I will use Azure CLI, just to show off my JSON filter queries. Note how the prefixes of the “indirect spokes” are not known to any of the virtual hubs:
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n $hub1_name -o table --query 'value[].{Address:addressPrefixes[0], NextHopType:nextHopType, ASPath:asPath}'
Address NextHopType ASPath
----------- -------------------------- -----------
10.1.1.0/24 Virtual Network Connection
10.2.1.0/24 Remote Hub 65520-65520
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub2_default_rt_id -g $rg -n $hub2_name -o table --query 'value[].{Address:addressPrefixes[0], NextHopType:nextHopType, ASPath:asPath}'
Address NextHopType ASPath
----------- -------------------------- -----------
10.1.1.0/24 Remote Hub 65520-65520
10.2.1.0/24 Virtual Network Connection
Since both hubs only know the prefixes for the VNets directly connected to Virtual WAN, but not those of the spokes “indirectly” connected, connectivity will not work with the default configuration, and you will need to configure some static routes here and there.
Static routes in Virtual WAN
Firstly, you need to understand static routes in Virtual WAN, since there are two types of them:
- Static routes in virtual hub route tables.
- Static routes in virtual network connections.
This is not obvious to portal users, because when you create static routes in a route table, the Azure portal does configure both types of routes via the “Configure” button for the “Next Hop IP”:
What this does is actually configuring two routes. The first one in the route table, where the next hop is the resource ID for the VNet connection (JSON-formatted for clarity):
❯ az network vhub route-table route list -n defaultRouteTable -g $rg --vhub-name $hub1_name -o jsonc
[
{
"destinationType": "CIDR",
"destinations": [
"10.1.8.0/21"
],
"name": "spokes11",
"nextHop": "/subscriptions/xxx/resourceGroups/testvwan/providers/Microsoft.Network/virtualHubs/hub1/hubVirtualNetworkConnections/spoke11",
"nextHopType": "ResourceId"
}
]
After packets arrive at the connection, the second route makes sure that the next hop is the IP address you specified:
❯ az network vhub connection show -n spoke11 --vhub $hub1_name -g $rg --query 'routingConfiguration.vnetRoutes.staticRoutes[].{Name:name, Prefix:addressPrefixes[0], NextHop:nextHopIpAddress}' -o table
Name Prefix NextHop
----------------- ----------- ---------
spokes11-cx-route 10.1.8.0/21 10.1.1.4
When you create routes via Terraform, PowerShell or Azure CLI you need to be aware of this dicotomy, because routes at the route table and VNet connection levels are independent from each other. It is only the Azure portal that offers the illusion of these static routes being the same.
The traditional way: static routes everywhere
The traditional way of enabling the “indirect spokes” setup consisted in using static routes at both levels (route tables and virtual network connections), as the following picture shows:
This is the setup documented in the official docs here. Looking at the static routes in each hub route table and each VNet connection, you would have something like this:
❯ az network vhub route-table route list -n defaultRouteTable -g $rg --vhub-name $hub1_name -o table
DestinationType Name NextHop NextHopType
----------------- -------- -------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------
CIDR spokes11 /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/testvwwan/providers/Microsoft.Network/virtualHubs/hub1/hubVirtualNetworkConnections/spoke11 ResourceId
CIDR spokes21 /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/testvwwan/providers/Microsoft.Network/virtualHubs/hub2/hubVirtualNetworkConnections/spoke21 ResourceId
❯ az network vhub route-table route list -n defaultRouteTable -g $rg --vhub-name $hub2_name -o table
DestinationType Name NextHop NextHopType
----------------- -------- -------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------
CIDR spokes21 /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/testvwwan/providers/Microsoft.Network/virtualHubs/hub2/hubVirtualNetworkConnections/spoke21 ResourceId
CIDR spokes11 /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/testvwwan/providers/Microsoft.Network/virtualHubs/hub1/hubVirtualNetworkConnections/spoke11 ResourceId
❯ az network vhub connection show -n spoke11 --vhub $hub1_name -g $rg --query 'routingConfiguration.vnetRoutes.staticRoutes[].{Name:name, Prefix:addressPrefixes[0], NextHop:nextHopIpAddress}' -o table
Name Prefix NextHop
----------------- ----------- ---------
spokes11-cx-route 10.1.8.0/21 10.1.1.4
❯ az network vhub connection show -n spoke21 --vhub $hub2_name -g $rg --query 'routingConfiguration.vnetRoutes.staticRoutes[].{Name:name, Prefix:addressPrefixes[0], NextHop:nextHopIpAddress}' -o table
Name Prefix NextHop
----------------- ----------- ---------
spokes21-cx-route 10.2.8.0/21 10.2.1.4
With this configuration everything works, but it is kind of cumbersome, especially if you cannot easily summarize the indirect spoke prefixes: for every new spoke you need to add routes to every single hub.
Don’t forget any spoke
You might be tempted to create routes for each spoke only in the closest Virtual WAN hub. After all, shouldn’t the hub advertise these routes to all the others? The answer is a clear no: when you configure a static route in a Virtual WAN hub route table, you are only configuring the static route for that specific hub, not a redistribution into BGP.
Consequently, if you happened to configure the route for spokes 111 and 112 only in hub 1, and spokes 211 and 212 only in hub 2, each hub would only know about its own spokes, but not those remotely connected:
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n $hub1_name -o table --query 'value[].{Address:addressPrefixes[0], NextHopType:nextHopType, ASPath:asPath}'
Address NextHopType ASPath
----------- -------------------------- -----------
10.1.8.0/21 Virtual Network Connection
10.1.1.0/24 Virtual Network Connection
10.2.1.0/24 Remote Hub 65520-65520
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub2_default_rt_id -g $rg -n $hub2_name -o table --query 'value[].{Address:addressPrefixes[0], NextHopType:nextHopType, ASPath:asPath}'
Address NextHopType ASPath
----------- -------------------------- -----------
10.2.8.0/21 Virtual Network Connection
10.1.1.0/24 Remote Hub 65520-65520
10.2.1.0/24 Virtual Network Connection
So here routing between the indirect spokes wouldn’t work at all, and your environment would be broken. Again here the lesson learnt is that this configuration requires static routes for all spokes in every single hub.
Redistributing static routes into BGP
However, since some months you can redistribute connection static routes into BGP. You configure the routes at the virtual network connection level, and let those routes be dynamically advertised throughout your network. This is what this toggle looks like in the Azure portal:
If you enable this “Propagate static route” feature, it means that no static route will be required any more in the route table, since the VNet connection routes will be redistributed into BGP:
Even without any static route at the route table level, the effective routes show that the indirect spoke prefixes are learnt via BGP:
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n $hub1_name -o table --query 'value[].{Address:addressPrefixes[0], NextHopType:nextHopType, ASPath:asPath}'
Address NextHopType ASPath
----------- -------------------------- -----------
10.1.1.0/24 Virtual Network Connection
10.1.8.0/21 Virtual Network Connection
10.2.8.0/21 Remote Hub 65520-65520
10.2.1.0/24 Remote Hub 65520-65520
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub2_default_rt_id -g $rg -n $hub2_name -o table --query 'value[].{Address:addressPrefixes[0], NextHopType:nextHopType, ASPath:asPath}'
Address NextHopType ASPath
----------- -------------------------- -----------
10.1.8.0/21 Remote Hub 65520-65520
10.1.1.0/24 Remote Hub 65520-65520
10.2.1.0/24 Virtual Network Connection
10.2.8.0/21 Virtual Network Connection
Traffic will flow between the spokes, without having to configure any static route at the route table level. If you happened to add a third virtual hub to the Virtual WAN environment, you wouldn’t have to add anything to its route table either.
Does this work with Routing Intent?
Not yet. Unfortunately, propagating VNet connection static routes is not supported at the time of this writing, as the Routing Intent documentation highlights in its Limitations paragraph:
Static routes on the Virtual Network connection with “static route propagation” aren’t applied to the next-hop resource specified in private routing policies. Support for applying static routes on Virtual Network connections to private routing policy next-hop is on the roadmap.
However, note that if your NVAs support BGP you can inject routes into Virtual WAN with dynamic routing. Unfortunately, that excludes appliances such as the Azure Firewall.
Conclusion
This was another geeky post of a technology that has been out there for a while with the slight twist of a new feature that is not supported yet in every scenario. Hopefully this write-up helped you to understand Virtual WAN a bit better. Please let me know in the comments!