Editor’s note: This is the second in an ongoing series on moving our network to the cloud internally at Microsoft.

At Microsoft, the Microsoft Digital Employee Experience (MDEE) team—our company IT organization—is using the Azure SDK, Azure Container Instances, and the Azure Compute Gallery to create a platform for deploying our virtual labs into secure, user-defined hub-and-spoke networks in Microsoft Azure. These labs provide isolated environments where our employees can create their own on-demand, scalable virtual machine and network environments for testing and development purposes.

This collection of technologies enables our employees to create virtual lab environments across multiple Azure tenants at scale, using infrastructure as code (IaC) to quickly deploy lab templates using the Azure Compute Gallery.

[Read the first blog in our “Moving our network to the cloud” series.]

ACI for flexibility and scalability

Azure Container Instances (ACI) is a critical component of our provisioning process. ACI is a fully managed service offered by Azure that enables users to deploy and run containerized applications in the cloud without having to manage virtual machines or learn new tools. It offers exceptional flexibility and scalability, making it ideal for managing our virtual labs environment.

ACI enables simplified orchestration of containers, especially when compared to more complex solutions like Kubernetes. ACI offers simple configuration for isolated containers, eliminating the need for deep knowledge of the network stack and the need to create complex YAML-based configurations. This simplicity streamlines the development process, reduces complexity, and ensures that container security measures are always included.

ACI also supports a wide variety of container images, including Docker containers and containers from other sources, such as Azure Container Registry, Docker Hub, or private container registries. In our experience, it scales very well with lightweight .Net Core images.

ACI offers rapid container deployment and orchestration. Our containers are available quickly to coordinate virtual lab deployment and can be shut down promptly when their work is completed. This dynamic allocation ensures that resources are only utilized when necessary. This works well in our stateless workload scenarios and is especially useful for batch processing. It also eliminates the overhead of cluster management tasks and lets us focus on deploying containers immediately.

We configure ACI to ensure graceful region-based failover. ACI offers versatile options for region failover and makes our business continuity and disaster recovery scenarios simple to implement. We use an Azure function to initialize failover groups based on region availability, creating a seamless user experience.

We use ACI for data processing, batch jobs, and event-driven functions where the workload varies and can be executed independently from the API services. We use messaging queues like Azure Service Bus to coordinate between the APIs running in Azure Kubernetes Service (AKS) and the background processing tasks in ACI. This configuration ensures that the API services can trigger or communicate with the background processing components when necessary.

Due to its ability to scale horizontally and quickly spin up instances without delay, we could continue delivering high performance to our users, even during heavy loads on our system. Our platform creates almost 40 thousand ACI instances each month.

The dynamic nature of ACI ensures that the resources are only utilized when necessary, keeping costs at a minimum. Additionally, we initialize containers with the fewest vCPU and memory resources required for their specific tasks to optimize resource allocation and cost tracking.

Getting started with containers can be intimidating, but ACI makes it very simple to deploy a container. With Hyper-V isolation by default, support for burst workloads, and a wide array of powerful capabilities, we can scale to the highest performance applications.

— Justin Song, senior software engineering manager, Azure Container Instances team

This fine-grained resource allocation ensures efficient utilization and simplifies cost tracking for each lab deployment, resulting in highly available, high-performing, cost-effective operations.

ACI’s serverless infrastructure allows developers to focus on developing their applications, not managing infrastructure. ACI provides the capacity to deploy containers and apply platform updates promptly to ensure security and compliance.

“Getting started with containers can be intimidating, but ACI makes it very simple to deploy a container,” says Justin Song, a senior software engineering manager on the Azure Container Instances team at Microsoft. “With Hyper-V isolation by default, support for burst workloads, and a wide array of powerful capabilities, we can scale to the highest performance applications.”

Azure Compute Gallery for rapid VM provisioning

We use the Azure Compute Gallery to bring efficiency and scalability to VM provisioning for our labs.

Azure Compute Gallery enables us to manage lab virtual machine images globally, with replication across multiple Azure regions.

Managed replication helps us ensure that VM images are readily available wherever our users need them. We’re also using custom least recently used (LRU) cache logic on top of the Gallery Image SDK to reduce the costs associated with hosting images across multiple regions. This custom logic ensures that unused replications are cleaned when not needed, reducing costs while still maintaining the accessibility and reliability of our virtual labs.

We allow our users to deploy pre-configured lab environments called templates. We can create versioned labs using Azure Compute Gallery’s versioning capabilities, effectively capturing unique lab configurations at different development stages. This feature enables our users to save and share meticulously crafted lab setups through templates, fostering global collaboration and knowledge sharing.

They can effortlessly create snapshots of their labs, simplifying collaboration, promoting consistency, and providing control over their virtual lab experiences. Azure Compute Gallery’s versioning puts lab management in the hands of our users, offering flexible, streamlined collaboration.

Role-based access control provides the core access management functionality for Azure Compute Gallery images. Using RBAC and Azure Active Directory identities, access to images and image versions can be shared or restricted to other users, service principals, and groups.

Azure SDK for efficient resource orchestration at scale

The Azure SDK for .NET provides the foundation for our platform’s scalability and resource management. We’re using the Azure SDK’s comprehensive set of open-source libraries, tools, and resources to simplify and expedite application and service development in Azure. The Azure SDK enables our development teams to ensure uniform features and design patterns for Azure applications and services across different programming languages and platforms.

Azure SDK packages adhere to common design guidelines—the Azure.Core package that is included in the SDK supplies a broad feature set, including HTTP request handling, authentication, retry policies, logging, diagnostics, and pagination. We’ve used the SDK to develop additional APIs that are easily integrated with other cloud-based services.

With the Azure SDK APIs, our developers have a unified interface to Azure services without needing to learn distinct APIs for each resource type. Development and resource management are streamlined across the entire Azure platform.

With a unified approach, we can use the Azure SDK to manage diverse resources across multiple Azure subscriptions and accounts.

Here are some tips for getting started with the Azure SDK, Azure Container Instances, and the Azure Compute Gallery at your company:

• Use ACI to simplify container orchestration with a smaller developer learning curve, especially when compared to more complex solutions like Kubernetes.
• Configure region failover using resources across multiple Azure regions to quickly deploy containers in healthy regions when another region fails. This ensures service continuity and provides a seamless experience for users.
• Use ACI scaling to quickly deploy instances across Azure regions, delivering high performance and availability for heavy loads systems.
• Configure replication in Azure Compute Gallery to provide global replication management for virtual machine images, ensuring images are readily available to users worldwide.
• Use Azure Compute Gallery versioning capabilities to allow users to capture unique virtual machine configurations at different development stages.
• Access important resources that can help you navigate this process with the Azure SDK. The Azure.Core package in the SDK offers a unified, standardized approach to accessing Azure functionality across various resource types.
• Use the Azure SDK to enable seamless management and deployment of data plane resources at scale across different Azure subscriptions and accounts.

Try out managed identities with Azure Container Instances.

• Read the first blog in our “Moving our network to the cloud” series.
• Check out these best practices and considerations for managing your Azure Container Instances.
• Learn how to manage your Azure security baseline for your Azure Container Instances.
• Discover more about cloud infrastructure and agility at Microsoft.

Please share your feedback with us—take our survey and let us know what kind of content is most useful to you.

The post Building a secure and efficient self-service application using Azure ACI, Azure Compute Gallery, and the Microsoft Azure SDK appeared first on Inside Track Blog.

Critical data and high risk environments

At MDEE, we classify approximately one percent of the services and data that we host as High Value Assets (HVAs). An HVA is a single isolated environment that provides a secure space for company workloads. Access to HVA data by unauthorized users could negatively affect Microsoft business in a significant way.

In our organization, we host several HVAs for different business groups that need a highly secure environment to prevent unauthorized access or data leaks. Most data in an HVA is classified as highly confidential. HVAs also host data that’s regulated by government policy or other legal restrictions, or that’s physically isolated from other datacenter assets and from our corporate network. A typical HVA can be broken down into several components:

• HVA fabric is the hosting environment for all HVAs. The HVA fabric encompasses the secure hosts, storage, computing, and network services used by all our internal HVA customers.
• HVA stamp is a single instance of an HVA that’s hosted on the HVA fabric. HVA stamps are also called HVA instances.
• Tier 0 holds highly privileged Active Directory resources (such as computer and user accounts) that can give an attacker significant access to the network. These resources include domain controllers, which host the Active Directory database, and highly privileged accounts or groups, such as domain admins or enterprise admins.
• Tier 1 hosts privileged services and systems used by HVAs. These resources provide control over enterprise servers and applications. Tier 1 contains a significant amount of business value hosted in the assets.
• Tier 2 is also called the customer tier. It hosts services that are provided and managed by the internal customer who uses the HVA. Each HVA hosts a unique set of customers and services. Tier 2 services may be privileged in comparison to other systems, but within the scope of the specific HVA, they are the least-privileged services.

HVA topology

A standard HVA host includes the three-tier administrative model and uses the HVA fabric for storage, network, and related services. The components of an HVA are distributed and managed in highly secured datacenters. Each access tier gives a layer of protection against credential theft.

The HVA system is multi-tenant. Each HVA stamp is an isolated environment that’s built for a specific customer or isolated workload. We use isolation techniques to help create clear boundaries between HVA stamps. HVA stamps can be of mixed size (with a different number of virtual machines, different sizes of virtual machines, and so on) and can host a variety of environments. One HVA stamp might host a single Tier 2 service, and others might host full end-to-end environments that have hundreds of servers.

The following figure shows a high-level view of an HVA environment with several HVA stamps.

Using shielded VMs for HVA

To create the private cloud environment that hosts our HVA resources, we use Windows Server 2019, System Center Virtual Machine Manager, and Windows Azure Pack (WAP). Windows Server 2019 introduces the shielded VM feature in Hyper-V. It protects virtual machines from threats outside and inside the fabric. It does this by encrypting disk and virtual machine states so that only virtual machine admins or tenant admins can access them.

By using System Center Virtual Machine Manager and Hyper-V host clusters in our private cloud environment, we can quickly and efficiently provision HVAs. We don’t have to worry about provisioning specific hardware to host HVA resources. The Windows Azure Pack offers a familiar, browser-based interface that our internal customers can use to provision resources. When needed, we provision shielded VMs and provide the computing resources to host an HVA workload.

Guarded fabric health attestation and key release

Shielded VMs are part of the guarded fabric system in Windows Server 2019 Hyper-V. The guarded fabric consists of several layered components:

• Code and boot integrity uses virtualization-based security to allow only approved code to run on the Hyper-V host from the moment it starts.
• Virtualization-based security uses hardware security technology in Windows Server 2019 to create an area that’s isolated from the Windows kernel and other applications to prevent external attacks.
• Trusted Platform Module (TPM) 2.0 is used to securely measure a Hyper-V host’s boot process and code integrity policy. These are then sent to the HGS as part of the health attestation process.
• Host Guardian Service (HGS) acts as an arbitration point for the guarded fabric that contains shielded VMs. HGS provides health attestation for the Hyper-V hosts and key protection for the material that’s required to run the shielded VMs.

Guarded host attestation

As illustrated in the figure below, HGS handles the attestation process for the guarded Hyper-V hosts on which the shielded VMs reside, including key requests and health information. This process ensures the health of the host, the protection of the shielded VM, and the appropriate access for users.

The attestation process includes the following steps:

1. The guarded Hyper-V host sends a key request to the HGS.
2. The HGS replies that it can’t verify that the Hyper-V host is a legitimate host.
3. The Hyper-V host sends its endorsement key to HGS from its TPM module to establish identity, along with health baseline and code integrity policy.
4. If HGS recognizes the identity of the Hyper-V host and considers the baseline and code integrity policy healthy, it supplies a certificate of health to the Hyper-V host.
5. The Hyper-V host re-sends the key request and health certificate to the HGS.
6. The HGS sends an encrypted response back to the Hyper-V host’s virtualization-based security, and the response can be decrypted only by the host hardware security module, to start the shielded VM.

Implementing HVA fabric using shielded VMs

The implementation of HVAs using shielded VMs starts at the datacenter. All HVA servers should be in physically isolated and secure environments. Physical access to the datacenter requires two-person access, and it’s limited to the HVA fabric team and the administrative team.

Physical access implementation

Best practices for implementing physical security components for the HVA include:

• Physical access to the hosting fabric hardware and datacenter floor should require two-person biometric access controls and smart card access to all server cages and racks.
• Physical access to the hosting fabric hardware and datacenter floor by an HVA team admin should require datacenter access tool tickets and a fabric admin escort.
• Datacenter floor access should be granted to only permanent employees.
• Cameras should be used to record all physical access to the datacenter floor and racks.
• The datacenter should have around-the-clock security guards on site—they monitor the facility, datacenter floor, and all access paths.

Hardware implementation

We use only specifically configured hardware in our HVA fabric. Our host hardware runs Windows Server 2019 and Hyper-V. The following table lists the components and management responsibilities.

Hardware components and management responsibilities

The following graphic shows the HVA hosting fabric, including HGS, for two primary sites.

The architecture groups together pods of Hyper-V servers as pods, managed by System Center Virtual Machine Manager and fabric domain controllers. The pods are controlled by a group of HGS servers, with access controlled by hardware security modules. We can use this layer separation to separate the administrators of the underlying virtualization fabric from the administrators of the applications and the administrators of the HGS.

Moving forward with HVA using shielded VMs and HGS

We’re experiencing several significant achievements in our HVA environment by using shielded VMs and HGS:

• Dedicated hosting environment (HVA fabric) is a separate host environment for HVAs. It includes specialized configuration and staff who manage the day-to-day operations of all fabric systems. This hosting environment uses the latest and greatest security configurations and systems to help protect HVA customers and workloads.
• HVA stamps isolation is a dedicated environment that holds all necessary services to support a designated workload. This isolated HVA stamp is built to protect itself from all outside threats.
• Role separation keeps every tier of access throughout the fabric, HVA stamps, and related systems isolated. It provides role separation between admins of different functions or types. Fabric admins, network admins, and HVA stamp admins all have isolated credentials and access rights. Most systems also require at least two-person access to change the HVA hosting fabric.
• Privilege elevation mitigation requires different credentials and remote access systems for each tier. IPsec, Group Policy, and silos prevent tier access cross-pollination.
• Network isolation keeps each HVA stamp isolated. All inter-network connections are terminated through a physical firewall. Intra-network connections are protected by IPsec, Windows Firewall, and Group Policy configuration.

• Microsoft extends Azure management to the private cloud with Azure Arc
• Migrating System Center Configuration Manager on-premises infrastructure to Microsoft Azure
• Step by Step – Configuring the Host Guardian Service in Windows Server 2016 – Microsoft Tech Community

The post Using shielded virtual machines to help protect high-value assets appeared first on Inside Track Blog.