Azure ExpressRoute Explained
In this article, we will explain Azure ExpressRoute. First, we will take a look at how ExpressRoute enables a private connection to the Microsoft backbone network, before moving on to the available SKUs and different types of ExpressRoute offerings, as well as how to enable resiliency. It’s everything you wanted to know but were afraid to ask!
Azure Network & Backbone
Microsoft has a huge global, low-latency backbone network that connects to the multiple express route gateways available for consumption by the Azure regions. Those Azure regions each consist of multiple datacenters (or zones), which connect to the express route gateways. As multiple gateways onto the backbone network are connected to each Azure datacenter, this forms a highly resilient connection to each Azure region. The diagrams below explain this more clearly:
Within an Azure region, multiple services exist which attach to the Microsoft backbone, for example, IaaS services like virtual machines, PaaS services like Azure App Services, and SaaS offerings like Office 365 that live in the Microsoft cloud rather than a specific region in Azure, but still connects to the Microsoft backbone network.
The Backbone network also extends to ‘edge nodes’. Services on the ‘edge’ include global services like Azure Front Door and Azure CDN. Edge nodes live in a carrier-neutral facility that different carriers can connect to also (think AT&T, Verizon, Vodafone, BT, etc.), which effectivity connect to and form the internet. This is how the Azure regions connect to the internet.
Also on the backbone network, are peering points (or ‘meet me’ locations) that facilitate private connectivity to customer networks. These are secure carrier-neutral locations that include lots of routers, forming the ‘Microsoft Enterprise Edge’ (or MSEE). Partners and service providers also have routers that interconnect to the MSEE to form a connection between the two. Any new Azure regions will have a peering point located close by, but this will not be part of the Azure region.
Finally, on the customer network, in the example of a physical location, fiber would be laid to the office by a service provider, and connected to the customer network using a router. MPLS networks (any-to-any) can also be connected to the service providers peering point.
When a connection is created, 2 connections are actually created for resilience. 2 BGP sessions are created to exchange routes. The 2 customer routers, connect to the 2 service provider routers, which connect to the MSEE routers, which then connect to Azure. Any 802.1q tags used for VLANs on premises are stripped out, as VLANs cannot be extended to Azure.
SKUs
Connecting to the backbone does not connect you to a particular Azure region. It connects you to the global backbone network. Which regions you can access is controlled by the SKU (or type of circuit) of the ExpressRoute connection you choose.
Local SKU
This allows you only to talk to the region local to the Peering point (i.e. the closest Azure region). e.g. If I connected an on-premise network to the London Peering point to the UK South Azure region this would be fine, but I could not connect to the UK West or US East Azure regions.
Some Peering points do not have the Local SKU available, e.g. Dallas, Atlanta to name a couple. To find out which Peering points have the Local SKU available and which Azure region they line up with, you can check the Microsoft Learn page on ExpressRoute partners and peering locations.
Standard SKU
This allows you to connect to any geo-political region within the peering point region. e.g. If I connected an on-premise network to the London Peering point to UK South or UK West Azure regions, this would be fine, but I could not connect to the US East region.
Premium SKU
This allows you to connect to other regions outside of the peering point region. e.g. If I connected an on-premise network to the London Peering point it could be connected to the US East Azure region.
With ExpressRoute, Ingress is free. Egress is metered. There is also an unmetered SKU that costs a lot more.
VNet Gateway SKUS
There are different SKUs for the VNet Gateway based on performance, using metrics such as speed, number of packets, and connections per second.
ExpressRoute Direct
ExpressRoute direct is an option for customers that want higher bandwidth connections. It provides dual 100-Gbps or 10-Gbps connectivity, that supports Active/Active connectivity at scale. It achieves this by connecting the on-premises router to the MSEE router directly (not through a service provider), directly connecting to 2 ports on the router. At the peering location, this means there is a physical (layer 1) connection directly to the MSEE routers ports.
One advantage of ExpressRoute direct, is the ability to use Media Access Control security (MACsec). This allows you to encrypt the traffic between the routers if desired, and therefore the traffic is encrypted on any physical cable the data travels within the Peering location to the MSEE router.
Circuits can then be created as required over the connection (e.g. If I have a 100Gbps ExpressRoute direct connection, I can split it up into 2x 50Gbps connections, 4x 25Gbps connections, or 1 x 50Gpbs and 2 x 25Gbps connections for example).
ExpressRoute Fast Path
Fast path can only be turned on with the UltraPerformance VNet Gateway SKU. This will enable traffic to bypass the VNet Gateway, allowing faster connections to bypass the Gateway that might cause a bottleneck. For example, if I had a 100Gps connection, and my Gateway only supported 10Gps, I would not be getting the full 100Gbps into my VNet. Fast path solves this issue, whilst also reducing latency.
Active Active
ExpressRoute connections are active-active for resilience, meaning you have 2 active connections. If you have a 1Gbps connection, you actually have 2Gbps because of the active-active nature of ExpressRoute.
Peering Types
Once the ExpressRoutehas been established, we need to create Expressroute circuits. Each circuit has a service key or S-Key.
There are 3 types of Peering that can be used over ExpressRoute.
Private Peering is used to connect to your private IP space in Azure. To set this up you will need 2 x /30 private IP address ranges set up in the on-premise network (you can also use public IP ranges but private ranges are recommended so as to not unnecessarily waste public addresses).
A VNet gateway will need to be created of type ‘ExpressRoute’ in its own subnet in your Azure Subscription. The minimum subnet size is a /28. If you want to also use the gateway for VPN connections as well as ExpressRoute for backup, then a /27 is recommended. More on that later.
Some gateways depending on the Azure region may span availability zones, or may use a single zone. For example, UK South gateways span 3 availability zones, whereas UK West gateways would only have a single zone.
Once the circuit has been created in the Azure Portal, you can get the resource ID of the circuit and generate an authorization ID (service key or S-Key). As the size of the SKU increases, more connections are allowed over the circuit. Multiple VNets can be connected to the circuit by connecting them together with VNet peering and a hub and spoke model to a central VNet gateway, rather than creating separate VNet gateways and circuits per VNet. This also avoids the cost incurred for using multiple VNet gateways. Inbound traffic will go via the VNet gateway, outbound traffic will not as there are other IP addresses it will need to talk to and it will also be used to enable Azure to learn get BGP routes.
Microsoft peering (used to connect to SaaS services not living within an Azure VNet such as Office 365 or those services in public address space). Office365 was designed to work over the internet, and as such is generally not recommended to be enabled unless in specific circumstances. You have to have Expressroute Premium for this to work. The Microsoft team must also whitelist you before you can enable this. To enable this, you need to have verified ownership of a /29 range of public IP address space. This will be split into 2x /30 ranges, giving 2 usable IPs each for each of the 2 connections. On the internal on-premises network, traffic will need to be NATTed with a separate range of public IPs that is not advertised to the internet to avoid Asymmetric routing (i.e. traffic is sent over the ExpressRoute to Azure and is returned via the internet, and likely blocked by your firewall).
Your network has an identifier called an Autonomous system number (ASN) which will need to be publically registered. However, like IP addresses, there are ranges of private ASNs reserved for internal use, which are 64512–65514 and 65521–65534 that can also be used. Microsoft uses AS 12076 for Azure public, Azure private, and Microsoft peering. Microsoft reserves ASNs from 65515 to 65520 for internal use. Both 16 and 32-bit ASN numbers are supported.
To limit the consumption of Microsoft services to only those that are required, a route filter can also be created through the Azure Portal to allow only certain services or services in particular regions to be propagated over the ExpressRoute. The route filter can be linked to the ExpressRoute circuit. The ExpressRoute circuit is created in an Azure subscription. The route filter when enabled will apply to any ExpressRoute circuit, no matter which subscription the Azure circuit is held in.
In the background, each Azure service in each region will have a certain grouping of IP addresses and also a BGP community. BGP communities are groupings of IP prefixes tagged with a community value. The route filter limits which services can be accessed over the ExpressRoute as only certain routes are advertised through the BGP community.
The use of route filters and BGP communities is particularly useful when talking about multi-region setups with multiple ExpressRoutes, as you can tailor which ExpressRoute circuit will learn the routes to certain regions, and configuring paths using these to avoid traffic getting routed on sub-optimal paths through your on-premises networks to Azure.
Available BGP communities can be looked up on the ExpressRoute routing requirements page on Microsoft Learn.
AS Path Prepending can also be used to route traffic correctly in Microsoft peering if you have a publically registered ASN. As your network has an ASN number, pretending adds the same ASN number twice when advertising your IP range to make it appear as a longer route, forcing the traffic to the shorter route. This is useful when you have multiple ExpressRoute circuits connected to the same Gateway. For example, if my ASN number was 1234, and I had a network range of 192.168.1.0 over circuit one which was the optimal route, my route would read 1234 192.168.1.0. Using path prepending, I could add another route reading 1234 1234 192.168.1.0 on the secondary circuit, effectively telling the route table that there are 2 hops involved, which is a longer route than the first route.
You can also configure Weighting on each circuit to manually direct traffic. In the case where multiple circuits are connected to the same gateway, one could be assigned a priority of 100, and another a priority of 10. The one with the higher number will be used first. If the circuit with priority 100 went down or was unavailable, the circuit with a priority of 10 would be used.
For routing from on-premises routers to Azure VNets, local preference on the routers should be used. Routes to Azure address space will be assigned a preference value, similar to weighting on the ExpressRoute circuits.
Public peering has been discontinued (previously used to connect to Microsoft services). If you have had this type of setup, it has not been removed.
Both Microsoft and Private peering can be turned on together, or separately.
Expressroute Global Reach
Global reach enables connectivity between 2 on-premise locations over the Microsoft Backbone. This can act as redundancy for existing connections between on-premise locations, and complement your existing WAN setup.
ExpressRoute Gateways for dedicated HSMs
These are required because dedicated HSMs are totally isolated physical devices sitting in the Azure data center. Gateways are required to connect them to the Microsoft backbone network.
Coexisting ExpressRoute and VPN
You might want to create a VPN connection as a backup in case your ExpressRoute fails. This connection applies only to virtual networks linked to the Azure private peering path. There’s no VPN-based failover solution for services accessible through Azure Microsoft peering.
You can also configure a Site-to-Site VPN to connect to sites that aren’t connected through ExpressRoute.
There are some limitations to know about before setting this up. Coexistence in a dual-stack VNet is not supported. If you’re using ExpressRoute IPv6 support and a dual-stack ExpressRoute gateway, coexistence with VPN Gateway won’t be possible.
The GatewaySubnet in your VNet must also be larger than /27 in size, before you can go ahead and create another VNet Gateway in the same subnet for your VPN connections.
If the gateway is being created to coexist with an ExpressRoute gateway, choose any SKU except for Basic and the VpnType must be RouteBased.
The ExpressRoute is always chosen as the preferred path if it is active, but if it fails, traffic can be routed over the VPN connection instead. The on-premise routing should be set for the same behavior to avoid asymmetric routing.
Note that each virtual network gateway has an hourly compute cost, as well as the cost for egress data transfer, so even if it is not being used, it will still incur costs. The price is based on the gateway SKU that you specify when you create a virtual network gateway.
Configuration steps can be found on Microsoft Learn, as well as more details on the VPN Gateway SKUs.
Enabling IPv6 support for Private Peerings
To enable IPv6 support over ExpressRoute, be sure to first add IPv6 address space to your VNet and GatewaySubnet, prior to creating a new virtual network gateway.
You will need to provide a pair of /126 IPv6 subnets that you own for your primary link and secondary links. From each of these subnets, you’ll assign the first usable IP address to your router as Microsoft uses the second usable IP for its router.
Full configuration instructions can be found on Microsoft Learn.
Summary
In this article we have taken a deep dive look at ExpressRoute, covering how it works in the background, and the available options.
For more information, check out John Saville’s awesome explanation on Azure ExpressRoute (which I used to help with this article). You can also check out more official ExpressRoute documentation on Microsoft Learn. Cheers!
About the author
Jack Roper is an experienced IT professional, focused on cloud technologies and DevOps. Specialising in Azure, Azure DevOps, Terraform & Kubernetes, Jack currently works at Avanade as an Azure Technical Lead in the Infrastructure Talent Community. If you enjoy the content you can read more on Medium – or feel free to buy Jack a beer!
