Editor’s note: This is the second in an ongoing series on moving our network to the cloud internally at Microsoft.

At Microsoft, the Microsoft Digital Employee Experience (MDEE) team—our company IT organization—is using the Azure SDK, Azure Container Instances, and the Azure Compute Gallery to create a platform for deploying our virtual labs into secure, user-defined hub-and-spoke networks in Microsoft Azure. These labs provide isolated environments where our employees can create their own on-demand, scalable virtual machine and network environments for testing and development purposes.

This collection of technologies enables our employees to create virtual lab environments across multiple Azure tenants at scale, using infrastructure as code (IaC) to quickly deploy lab templates using the Azure Compute Gallery.

[Read the first blog in our “Moving our network to the cloud” series.]

ACI for flexibility and scalability

Azure Container Instances (ACI) is a critical component of our provisioning process. ACI is a fully managed service offered by Azure that enables users to deploy and run containerized applications in the cloud without having to manage virtual machines or learn new tools. It offers exceptional flexibility and scalability, making it ideal for managing our virtual labs environment.

ACI enables simplified orchestration of containers, especially when compared to more complex solutions like Kubernetes. ACI offers simple configuration for isolated containers, eliminating the need for deep knowledge of the network stack and the need to create complex YAML-based configurations. This simplicity streamlines the development process, reduces complexity, and ensures that container security measures are always included.

ACI also supports a wide variety of container images, including Docker containers and containers from other sources, such as Azure Container Registry, Docker Hub, or private container registries. In our experience, it scales very well with lightweight .Net Core images.

ACI offers rapid container deployment and orchestration. Our containers are available quickly to coordinate virtual lab deployment and can be shut down promptly when their work is completed. This dynamic allocation ensures that resources are only utilized when necessary. This works well in our stateless workload scenarios and is especially useful for batch processing. It also eliminates the overhead of cluster management tasks and lets us focus on deploying containers immediately.

We configure ACI to ensure graceful region-based failover. ACI offers versatile options for region failover and makes our business continuity and disaster recovery scenarios simple to implement. We use an Azure function to initialize failover groups based on region availability, creating a seamless user experience.

We use ACI for data processing, batch jobs, and event-driven functions where the workload varies and can be executed independently from the API services. We use messaging queues like Azure Service Bus to coordinate between the APIs running in Azure Kubernetes Service (AKS) and the background processing tasks in ACI. This configuration ensures that the API services can trigger or communicate with the background processing components when necessary.

Due to its ability to scale horizontally and quickly spin up instances without delay, we could continue delivering high performance to our users, even during heavy loads on our system. Our platform creates almost 40 thousand ACI instances each month.

The dynamic nature of ACI ensures that the resources are only utilized when necessary, keeping costs at a minimum. Additionally, we initialize containers with the fewest vCPU and memory resources required for their specific tasks to optimize resource allocation and cost tracking.

Getting started with containers can be intimidating, but ACI makes it very simple to deploy a container. With Hyper-V isolation by default, support for burst workloads, and a wide array of powerful capabilities, we can scale to the highest performance applications.

— Justin Song, senior software engineering manager, Azure Container Instances team

This fine-grained resource allocation ensures efficient utilization and simplifies cost tracking for each lab deployment, resulting in highly available, high-performing, cost-effective operations.

ACI’s serverless infrastructure allows developers to focus on developing their applications, not managing infrastructure. ACI provides the capacity to deploy containers and apply platform updates promptly to ensure security and compliance.

“Getting started with containers can be intimidating, but ACI makes it very simple to deploy a container,” says Justin Song, a senior software engineering manager on the Azure Container Instances team at Microsoft. “With Hyper-V isolation by default, support for burst workloads, and a wide array of powerful capabilities, we can scale to the highest performance applications.”

Azure Compute Gallery for rapid VM provisioning

We use the Azure Compute Gallery to bring efficiency and scalability to VM provisioning for our labs.

Azure Compute Gallery enables us to manage lab virtual machine images globally, with replication across multiple Azure regions.

Managed replication helps us ensure that VM images are readily available wherever our users need them. We’re also using custom least recently used (LRU) cache logic on top of the Gallery Image SDK to reduce the costs associated with hosting images across multiple regions. This custom logic ensures that unused replications are cleaned when not needed, reducing costs while still maintaining the accessibility and reliability of our virtual labs.

We allow our users to deploy pre-configured lab environments called templates. We can create versioned labs using Azure Compute Gallery’s versioning capabilities, effectively capturing unique lab configurations at different development stages. This feature enables our users to save and share meticulously crafted lab setups through templates, fostering global collaboration and knowledge sharing.

They can effortlessly create snapshots of their labs, simplifying collaboration, promoting consistency, and providing control over their virtual lab experiences. Azure Compute Gallery’s versioning puts lab management in the hands of our users, offering flexible, streamlined collaboration.

Role-based access control provides the core access management functionality for Azure Compute Gallery images. Using RBAC and Azure Active Directory identities, access to images and image versions can be shared or restricted to other users, service principals, and groups.

Azure SDK for efficient resource orchestration at scale

The Azure SDK for .NET provides the foundation for our platform’s scalability and resource management. We’re using the Azure SDK’s comprehensive set of open-source libraries, tools, and resources to simplify and expedite application and service development in Azure. The Azure SDK enables our development teams to ensure uniform features and design patterns for Azure applications and services across different programming languages and platforms.

Azure SDK packages adhere to common design guidelines—the Azure.Core package that is included in the SDK supplies a broad feature set, including HTTP request handling, authentication, retry policies, logging, diagnostics, and pagination. We’ve used the SDK to develop additional APIs that are easily integrated with other cloud-based services.

With the Azure SDK APIs, our developers have a unified interface to Azure services without needing to learn distinct APIs for each resource type. Development and resource management are streamlined across the entire Azure platform.

With a unified approach, we can use the Azure SDK to manage diverse resources across multiple Azure subscriptions and accounts.

Here are some tips for getting started with the Azure SDK, Azure Container Instances, and the Azure Compute Gallery at your company:

• Use ACI to simplify container orchestration with a smaller developer learning curve, especially when compared to more complex solutions like Kubernetes.
• Configure region failover using resources across multiple Azure regions to quickly deploy containers in healthy regions when another region fails. This ensures service continuity and provides a seamless experience for users.
• Use ACI scaling to quickly deploy instances across Azure regions, delivering high performance and availability for heavy loads systems.
• Configure replication in Azure Compute Gallery to provide global replication management for virtual machine images, ensuring images are readily available to users worldwide.
• Use Azure Compute Gallery versioning capabilities to allow users to capture unique virtual machine configurations at different development stages.
• Access important resources that can help you navigate this process with the Azure SDK. The Azure.Core package in the SDK offers a unified, standardized approach to accessing Azure functionality across various resource types.
• Use the Azure SDK to enable seamless management and deployment of data plane resources at scale across different Azure subscriptions and accounts.

Try out managed identities with Azure Container Instances.

• Read the first blog in our “Moving our network to the cloud” series.
• Check out these best practices and considerations for managing your Azure Container Instances.
• Learn how to manage your Azure security baseline for your Azure Container Instances.
• Discover more about cloud infrastructure and agility at Microsoft.

Please share your feedback with us—take our survey and let us know what kind of content is most useful to you.

The post Building a secure and efficient self-service application using Azure ACI, Azure Compute Gallery, and the Microsoft Azure SDK appeared first on Inside Track Blog.

The existing Microsoft Invoice platform allowed only manual invoice submission. External suppliers and internal users in Microsoft’s Accounts Payable (AP) Operations team could either email a scanned invoice or PDF, manually enter information into a web portal, or use that portal to bulk-upload invoices using a formatted Microsoft Excel template.

In some countries or regions with complex requirements, the AP Operations team is required to manually enter paper invoices directly into SAP, Microsoft’s financial system of record. The system worked, but it was inefficient.

Compared to the wider digital transformation at Microsoft, the inefficiency was glaring. Across the company, manual processes are being replaced with automated processes so that employees can focus on more impactful work. The Invoice Service team, which sits in the Microsoft Digital organization, saw an opportunity in the invoice processing system to modernize.

The goal was to trigger the creation of invoices using simple signals, like when purchased goods were received.

“We started with a question,” says James Bolling, principal group engineering manager for the Microsoft Invoice Service team. “How do we trigger invoices so that a supplier can just call our API and generate the invoice in a system-to-system call? How do we automate approval based on purchase order and invoice and receipting information?”

[Read about how we are digitizing contract management. Learn how we are using anomaly detection to automate royalty payments. Microsoft has built a modern service architecture for its Procure-to-Pay systems—read about it here.]

Lower costs, increased speed, and improved compliance

The Invoice Service team is responsible for the entirety of invoicing at Microsoft, Bolling says. The system it maintains integrates tightly with purchase orders related to goods and services from all Microsoft suppliers. The AP Operations team is tasked with ensuring that every payment adheres to relevant payment terms and service-level agreements.

The team also must maintain airtight compliance for the more than 120 countries and regions in which Microsoft conducts business, which accounts for about 1.8 million invoices per year and some USD60 billion in spend, according to Bolling.

The opportunity to lower operating costs by increasing speed and reducing the possibility of human error was enticing, but it wasn’t until the team began tackling a separate project that the scope of what it was about to undertake was clear.

Rewriting a 20-year-old legacy system

While working on a tax project, Shweta Udhoji and Guru Balasubramanian, both of them program managers on the Invoice Service team, spoke to customers and partners who used the legacy system regularly. Those conversations revealed the scale of the problem. Roughly 35,000 invoices were being submitted via email every month, with several thousand more coming in through the web portal.

Because validation paths are required for each intake method, they were present in duplicate or triplicate, creating redundancy that made it difficult to add a simple modification. Each change had to be applied to each method individually.

“The processes are more than 20 years old, and any extensions due to changing global compliance requirements or any other business needs that come in from our partner teams were very difficult to accommodate,” Udhoji says. “We wanted to simplify that.”

To make matters worse, the team couldn’t rely on documentation for a 20-year-old architecture as they looked for temporary fixes to get the time-sensitive tax updates out the door.

“We didn’t have any documentation to look at, so we had to test extensively, and once we started testing, we started seeing a lot of problems,” Udhoji says.

The road to the Modern Invoice API

Quick fixes wouldn’t solve the underlying problems of the legacy architecture. The team realized that it would need to be completely rewritten to adhere to modern standards.

The Modern Invoice API became a critical component of the broader effort to automate invoice creation and submission where possible. For partners and suppliers for whom manual intake methods are sufficient (or where paper invoices are required by law), those methods would largely remain intact, with some process optimizations added for efficiency. For Microsoft’s largest partners and suppliers, the API would automate the invoice creation and submission process.

“We knew we could make a huge impact on processing time and the manual effort required to process an invoice. We just needed to automate the process,” Udhoji says.

Because business needs change so much faster today than they did 20 years ago, the API was a business decision as well as a technical one. Modifications and extensions would need to be easy to add to keep up.

“What we were building with the Modern API was a framework for better automation, quicker changes, easier automation,” says Bryan Wilhelm, senior software engineer on the Modern Invoice API project.

Bridging legacy and modern systems

The challenge that the team faced was daunting and delicate. Because all invoice processing ran through the legacy architecture, there could be no interruptions in service—business must continue as usual, all over the world. The team also needed to be responsive to constantly shifting local compliance laws, too, adding modifications without downtime.

“We had to first understand the domain, the business requirements, and the technical side of it, while at the same time maintaining the legacy tool and thinking about how to re-imagine the invoice experience,” Balasubramanian says.

The team started by building a hybrid architecture model (as illustrated in the following diagram) on top of the legacy system; initially, the API would simply call the legacy invoice creation pipeline. By integrating with the existing process and building a wrapper on top of it, the legacy system would continue to function without interruption. With so many business rules and validation processes to consider, it would’ve taken a considerable amount of time to write an end-to-end process from scratch.

The iterative approach meant that the team could ship a working API, complete with integration with the legacy system, in just eight weeks. That left more time to gather and integrate early feedback from partners and suppliers while at the same time modernizing the underlying invoice processing pipeline.

Using Microsoft Azure and cXML for interoperability

The legacy system runs on Windows Server 2016 and SQL Server on-premises in Azure Infrastructure as a Service Virtual Machines, so the team leveraged SQL Datasync for synchronization between Azure SQL and on-premises SQL Server, and Azure Service Bus messaging for communication between the two systems. The API microservice was built as an Azure function.

For two reasons, the team chose to adopt the commerce eXtensible Markup Language (cXML) protocol to enable communication between documents. First, cXML is compliant with existing Microsoft business rules out of the box. “All the gaps we saw that were missing in the legacy system were accounted for in cXML,” Wilhelm says.

Second, cXML has a robust community; thus, extensive documentation and support already existed, including around the business rules inherent to the cXML protocol.

Automation’s immediate impact

The Modern Invoice API went live globally to internal partners in August, and to date USD85 million worth of invoices have been sent using the API. As the API project evolves to encompass a greater share of all invoice processing, that touchless invoice submission and approval will lower operating costs and eliminate inefficiencies both for internal teams and for Microsoft vendors and partners across the globe.

Prior to the API, the Microsoft Corporate, External, and Legal Affairs (CELA) team used the web portal in conjunction with an internal tool that tracks US work visa workflows. Microsoft sponsors a significant number of employees who are in the United States on work visas, and the CELA team tracks the status of those visas and submits payment to U.S. Citizenship and Immigration Services (USCIS).

The old process involved running a report with the team’s internal tool to find out which visas required payment. They then used that information to populate an Excel template and submit the invoice in the web portal. Because the team processes tens of thousands of checks per year, the time wasted in this process added up.

CELA became one of the first groups to fully implement integration with the API, modifying its internal tool to call the API and automatically submit checks to file USCIS cases. By managing the process end to end within the same system, the team has seen a reduction in the resources required to order daily checks and gained complete visibility into what checks are being ordered and why.

Modernizing for today and tomorrow

Like CELA, business partners and suppliers who submit thousands of invoices per year to Microsoft had to build and manually upload formatted Excel files to a web portal. In creating the foundation for the Modern Invoice API, the team also laid the groundwork for other automated invoice submission and creation methods such as the SAP Evaluated Receipt Settlement (ERS) process. In addition to adding support for 20 countries/regions that the legacy system simply didn’t (or couldn’t) support, these combined automation efforts mean that as much as 70 percent of the 1.8 million invoices submitted to Microsoft every year will be generated via automated, system-to-system calls.

New capabilities are in the pipeline, too. Supporting documentation can be attached to invoices using the new API, which wasn’t possible before. The team is also working on integrating Microsoft’s financial systems with those of GitHub, a recent Microsoft acquisition, to increase the speed of integration of future acquisitions. The API provides a simpler way to migrate the GitHub invoice system to Microsoft systems and components. “It would’ve been a crazy modification in the classic system,” Udhoji notes. Future acquisitions and integrations will be similarly affected as the API standardizes the process of invoice system migration.

In addition to the four tenants already using the Modern Invoice API, six more tenant groups are slated to be added by the end of the fiscal year, with an external rollout to Microsoft’s biggest external suppliers not far behind.

Creating, submitting, and approving more than 1.8 million invoice transactions every year required significant manual efforts for Microsoft and its payees. All told, the manual processes added up to 300,000 hours of work, according to Luciana Siciliano, Finance Operations director and global process owner.

“Our internal business groups no longer have to manually touch these invoices to get them into our enterprise invoice system and move them forward to complete the payout,” she says. “Through automated intake, receipting, and approval solutions, we can drive quick-turnaround solutions that are centered on our users.”

• Read about how we are digitizing contract management.
• Learn how we are using anomaly detection to automate royalty payments.
• Read about how Microsoft has built a modern service architecture for its Procure-to-Pay systems.

The post How retooling invoice processing is fueling transformation inside Microsoft appeared first on Inside Track Blog.