Our Platform Engineering team in Microsoft Digital Employee Experience (MDEE) wanted to improve the capabilities, performance, and resiliency of our on-premises integration platform. To do this, the team used Microsoft Azure Integration Services to build a cloud-based integration platform as a service (iPaaS) solution that increased data-transaction throughput and integration capabilities for our enterprise data footprint and improved platform reliability.
Business-to-business (B2B) and app-to-app (A2A) integration are imperatives in modern software solutions. Integration services use middleware technology that helps secure communication between integration points and data exchange between diverse enterprises and business applications. At Microsoft, our business demands integration across multiple independent software systems with diverse message formats such as EDIFACT, X12, XML, JSON, and flat file. Modern integration requires many modes of connectivity and data exchange, and includes the ability to connect:
- Two or more internal applications.
- Internal applications to one or more business partners.
- Internal applications to software as a service (SaaS) applications.
[Discover streamlining vendor assessment with ServiceNow VRM at Microsoft. Explore shining a light on how Microsoft manages Shadow IT. Unpack implementing a Zero Trust security model at Microsoft.]
Building on a foundation of enterprise integration
For decades, we as a company have worked to integrate our business data internally and in business-to-business scenarios with partners, vendors, and suppliers. BizTalk Server has been a standard for integration services for us and our partners, providing a foundation for dependable, easy-to-configure data integration.
Our ongoing digital transformation is driving cloud adoption to move business resources out of datacenters. As data storage and application development has evolved, cloud-native solutions based on SaaS and PaaS models have predominated among enterprise applications in most industries. To meet the growing need to supply increased scalability, reduce maintenance overhead for infrastructures, and decrease total cost of ownership, our Platform Engineering team has increasingly moved toward cloud-based solutions for enterprise integration.
Transforming integration with Microsoft Azure
Our Platform Engineering team began investigating Microsoft Azure Integration Services as a potential solution for scalable, cloud-based enterprise integration. Integration Services combines several Microsoft Azure services, including Logic Apps, API Management, Service Bus, Event Grid, and Azure Functions. These services provide a complete platform that companies can use to integrate business applications and data sources. Our team began working with Integration Services to gauge feasibility, test integration scenarios, and plan for enterprise-scale integration capabilities on the platform.
Collaborating to improve Microsoft Azure Integration Services
Throughout the development process, our Platform Engineering team worked closely with the Integration Services product group to enhance and build connectors. This collaboration allowed us to suggest improvements to existing Integration Services functionality. This effort prompted the creation of two new Logic Apps connectors—SAP with Secure Network Communication (SNC) and Simple Mail Transport Protocol (SMTP)—and enhancements to two existing Logic Apps connectors (EDIFACT and X12).
Examining our Azure Integration Services architecture
We in MDEE use all Microsoft Azure Integration Services components in its architecture to support end-to-end integration. Each component supplies an important part of the larger solution, including:
- API Management for APIs, policies, rate limiting, and authentication.
- Logic Apps for business workflows, orchestration, message decoding and encoding, schema validations, transformations, and integration accounts to store B2B partner profiles, agreements, schemas, and certificates.
- Microsoft Azure Event Grid for event-driven integration to publish and subscribe to business events.
- Microsoft Azure Functions for writing custom logic tasks, including metadata and config lookup, data lookup, duplicate check, replace namespace, and replace segments.
- Microsoft Azure Data Factory for processing low volume, large payload messages, ETL processes, and data transformation.
We used Microsoft Azure Front Door as the entry point for all inbound traffic and helped secure endpoints by using Microsoft Azure Web Application Firewall configured with assignment permissions for allowed IP addresses. Additionally, API Management enabled us to abstract the authentication layer from the processing pipeline to help increase security and simplify processing of incoming data.
We deployed the entire solution to an integration service environment, which supplied a fully isolated and dedicated integration environment and other benefits, including autoscaling, increased throughput limits, larger storage retention, improved availability, and a predictable cost model.
The following figure illustrates our solution’s architecture using Microsoft Azure Integration Services.
The solution architecture adheres to several important design principles and goals, including:
- Pattern-based workflows that enable dynamic decisions using partner information.
- Self-contained extensible workflows that can be modified and improved without affecting existing components.
- A gateway component to store and forward messages.
- Publish and subscribe services for data pipeline output.
- Complete B2B and A2A pipeline processing with 100 transactions per second throughput and message handling up to 100 megabytes (MB) per message.
Designing dataflow pipelines
Our dataflow pipelines perform processing for most of our business-data transformation and movement tasks. We designed the B2B and A2A processing pipelines using Logic Apps and Microsoft Azure Functions, processing documents in their native format and delivering them to line of business (LOB) or enterprise resource planning (ERP) systems such as Finance, HR, Volume Licensing, Supply Chain, and SAP.
- B2B pipeline. Electronic data interchange (EDI) documents such as purchase orders are brought in using AS2, processed using X12 standards, transformed, decoded and encoded using Logic Apps and Azure Functions, and then sent to the LOB app using the Logic Apps HTTP adapter.
- A2A pipeline. Documents such as XML/JSON come in using one of the built-in adapters including SAP, File, SQL, SSH File Transport Protocol (SFTP), or HTTP. The documents are debatched, transformed, decoded, and encoded using Logic Apps and Azure Functions, and then sent to the line-of-business system using the appropriate Logic Apps adapter.
Our integration solution used these pipelines in practical business scenarios across many lines of business at Microsoft, such as for volume licensing. A hardware manufacturer that includes Windows or Microsoft Office in their laptops submits an order for Windows or Office license to Microsoft’s ordering system, which sends the order details to our integration suite. The suite validates the messages, transforms them to IDoc format, and routes the IDoc to SAP using a data gateway for taxation and invoice generation. SAP generates an order acknowledgement in IDoc format and then passes it to the integration suite, which transforms the IDoc message into a format that the Microsoft ordering system will recognize.
Here’s another example from Microsoft Finance. An employee incurs an expense using a corporate credit card and the issuing financial institution sends a transaction report to the integration solution, which validates the message and performs currency conversion before sending it to Microsoft’s expense-management system for further approvals. After it’s approved in the expense-management system, the remittance transaction flows through the integration suite back to the banking system for payment settlement.
Capturing end-to-end messaging telemetry
We designed our solution to monitor message flow across the pipeline. Every transaction injects data into the telemetry pipeline using Microsoft Azure Event Hubs. The pipeline synthesizes and correlates that data to identify end-to-end processing status and recognize runtime failures. We built a custom tracking service that monitors and tracks important metrics for end-to-end workflows by using visual indicators on a dashboard. Accurate and readily available telemetry creates a more robust and reliable integration environment and improves the customer experience across pipelines.
We’ve realized several benefits across our integration environment, including:
- Increased scalability. Our integration solution processes millions of monthly transactions, including 10 million B2B, 2.5 million A2A, and 74 million hybrid cloud transactions.
- Improved quality of service. We used cross-region deployment with active-active configuration and thorough handling of faults to help achieve 99.9 percent in availability and reliability metrics.
- Reduced total cost of ownership. We’ve reduced monthly costs in Microsoft Azure by more than 40 percent with this iPaaS solution.
- Increased customer engagements. We’re working toward increasing Microsoft Azure Integration Services adoption by promoting this solution to our partners, vendors, and suppliers.
Microsoft Azure Integration Services has created an improved and more efficient integration environment for Microsoft. The increased scalability, reliability, and cost-effectiveness of Azure Integration Services has moved our business into a better position to actively collaborate with and operate alongside our partners, suppliers, and vendors. We’re continuing to transform our integration services landscape with Azure Integration Services to keep pace with the rapidly changing modern business environment.
