Now Microsoft Digital is running the majority of the company’s servers, internal apps, and business processes on Microsoft Azure in the cloud.

That move significantly reduced the volume of servers needed on site and made it easier to track the costs associated with running each service. It also helped IT administrators and developers apply standard risk-management policies and other best practices for network and data security.

But in some scenarios, Microsoft teams still use physical servers to fulfill specific needs.

Why is on-premises hardware still needed?

“Sometimes it’s the kind of app a team is using,” says Dana Baxter, a principal service engineer in Microsoft Digital’s Manageability division. “They might not have the infrastructure to move it to the cloud. There might be dependencies on systems that are not yet migrated.”

If a service is not going to be used for the long term, it is often not worth the effort of decommissioning, redesigning, and redeploying it. A handful of Microsoft groups also maintain on-premises servers because they require extremely high-speed direct internet connections.

At Microsoft, the Manageability Platforms team uses Microsoft System Center Configuration Manager (SCCM) for on-premises server management. In alignment with earlier IT design principles, SCCM covers only Windows servers, specifically those joined to a domain and assigned to the correct organizational unit (OU).

As Microsoft Digital began using more Infrastructure as a Service (IaaS) features native to Azure, a gap grew between the tools used to manage on-premises infrastructure and those used to manage IaaS. With some 3,000 on-premises servers still running within the Microsoft network infrastructure as of early 2019, Baxter saw a significant opportunity to improve the security, cost accounting, and manageability of these computing assets.

Microsoft customers face similar issues.

“Sometimes it doesn’t make sense to lift and shift everything immediately,” says Jian Yan, principal program manager for Azure Arc for servers. “If the hardware is paid for and it’s not at end-of-life, then it’s an investment they’ve already made.”

Baxter wanted Microsoft IT administrators to be able to manage the servers and virtual machines (VMs) with the same ease as an Azure dashboard. Could there be a way to connect these assets to Azure?

“About a year ago, the manageability team started working with the Azure product group on a vision for how we could replicate the functionality of SCCM using Azure features,” Baxter says.

The cross-group team was especially interested in supporting software deployment and collecting data for configuration settings across the organization. Another goal was to improve anti-malware measures for Microsoft Digital’s entire hybrid environment with a unified set of Azure features.

The team immediately realized many issues could be prevented or overcome by including on-premises servers in the Azure management tools. They decided to develop Azure solutions to cover the multi-OS platform and hybrid environments rather than expand usage of SCCM capabilities.

Enter Azure Arc, an extension of Azure Resources Manager, now in public preview. The service brings Azure features that are typically available only in the public cloud to private and on-premises workspaces, including those that are using non-Microsoft cloud services. It contains Azure Arc for server and Kubernetes management, and Azure data services.

With Azure Arc, IT administrators can use the Azure Control Plane to collect and view system data from any environment (on-premises, Azure) or platform (Windows, Linux). When the assets are visible to Azure, it is much easier to apply standard security policies and gather relevant information from each with automated cloud services.

Who is going to use Azure Arc on their servers?

“There are many use cases for Azure Arc. It gives us an opportunity to streamline and reduce the tools we use to manage infrastructure,” Baxter says. “For example, at the Azure Control Plane level, we now have a framework enabling enterprise IT security and governance admins to apply Azure policies at scale.”

For customers, Azure Arc for servers could help IT manage assets across more than one cloud provider. The service enables administration of non-Azure cloud servers alongside Azure assets.

“Users are going to different clouds to acquire their data,” Yan says. “It puts IT in a very difficult position. They need a way to consolidate all these different pieces and standardize across the organization.”

[Learn more about Microsoft’s cloud centric transformation, find out how the company adopted Azure monitor, and discover what principles to keep in mind when implementing modern engineering.]

Reports from early adopters

Microsoft Digital is now in the process of deploying Azure Arc for servers at Microsoft, beginning with the Managed Workspaces team. The rollout has just started, with roughly 10 percent of formerly isolated Microsoft servers and VMs becoming visible to Azure within the past few weeks.

Now, all Microsoft teams can implement enterprise-wide governance programs like management groups and policies that protect the entire company.

The use of Azure services is strategically important for both Azure and Microsoft.

“Azure Arc, Guest Configuration policy, Azure Policy, and Management Groups together allow seamless governance and management of on-premises and multi-cloud resources with a single control plane,” Baxter says.

Heathcliff Anderson, a service engineer in the Managed Workspaces group, was one of the first to try out the tool.

“We started slowly by rolling out the agent on machines one by one. Azure has a nice prescriptive guide on the website on how to do that installation,” Anderson says.

The team soon discovered that there was a point in the process of registering with Azure that required an IT admin to visit a website and manually enter a code. By using the Service Principal Name feature in Azure, Anderson was able to quickly develop a PowerShell script to complete this user action with automation.

“After testing the script on one or two machines, we launched the job through SCCM, running the script against about 100 servers at first. It took about 10 minutes,” Anderson says.

Today, the Managed Workspaces team has activated Azure Arc on more than 300 virtual and physical production servers and is running it with no issues. The servers now automatically receive and implement Azure Policies from the central governance teams in alignment with Azure IaaS systems.

Manasi Choudhari, a program manager in the Managed Workspaces group, is pleased with the benefits that the extension has delivered so far. The next step is to reduce the volume of manual IT administration for the Managed Workspace team.

“We hope to use the Azure extension for automation around deploying scripts that are needed for on-prem servers,” Choudhari says “It is very early, but these are very good features for us to explore.”

Other Microsoft teams also see the value of Azure Arc for servers.

“Tracking costs associated with on-prem servers has always been a difficult thing to do,” says Jeromy Statia, a principal software engineer responsible for securing the Windows Build pipeline. “We want to understand our resources and how they contribute to our services cost. An Azure subscription owner is very clear and defined. We know the cost of a service and who to go to when the server is not acting appropriately.”

Security policies are also easier to enforce with Azure Arc.

“In Azure, there’s this managed service identity that makes an app developer’s security management very easy,” Statia says. “It solves some of the worst practices and encourages best practices instead.”

The Managed Workspace team was able to provide specific product development input based on their experience so far with Azure Arc.

“The problem we’ve presented back to the product group,” Baxter says, “is right now, everyone has to download the package and connect it manually. How do we build this into the product so it’s set by default? How do we build the VM so it already has Arc Agent on it? We are asking the product team to make the agent more integrated.”

What’s next for Azure Arc for servers?

Having completed their initial rollout, the Managed Workspace team is anticipating the release of new Azure Arc capabilities.

“As extensions become available, we’ll run those and pilot those with the various groups,” Anderson says. “If we have any kind of configuration management policy changes that go out, now all of our security policies can be managed from Azure.”

Statia is especially looking forward to using Azure to support certificate auto-renewal. An Azure Key Vault Certificate Deployment extension (currently in Private Preview) keeps the certificate on any machine up to date.

“The reason I latched onto Arc Agent early was what I call the ‘bootstrap credential problem,’” Statia says. “Interacting with Azure always requires a pre-existing certificate. If you don’t already have a certificate, you need another method to get it.”

This could create a problem for users and require IT administrators to manually manage the certificates.

“With Azure,” Statia says, “I will no longer have to manage that credential for an on-premises server. We can use all the value-add of Azure in a standards-based way—soon, without having to worry about storing certificates with personal information exchange (PFX) files, the password that is managing PFX, or the deployment of the PFX package.”

In the future, the Azure product team plans to develop further inventory functionality for Azure Arc.

“The Manageability Platforms teams at Microsoft is creating an Azure-based Inventory solution, co-developed with the Azure product group, to replace our SCCM infrastructure,” Baxter says. “This will give us greater coverage and increase the breadth of data points we are able to collect.”

But this is just the beginning for Azure Arc.

“This is really an early stage of our journey,” Baxter says. “We are looking at expanding Azure Arc capabilities to leverage Azure Policy more widely.”

The team is also starting to support system configuration data collection across the entire Microsoft Digital environment for servers.

“The focus right now is around creating the foundation,” Baxter says. “We want to manage all our servers from Azure, so we can use the same tools for enterprise security and governance programs regardless of the asset’s location or operating system.”

Discover more about Azure Arc from the Microsoft Azure product group, including about About Azure Arc, Azure Arc for servers, and Azure’s Cloud Adoption Framework.

• Learn more about Microsoft’s cloud centric transformation.
• Find out how the company adopted Azure monitor.
• Discover what principles to keep in mind when implementing modern engineering.

The post Microsoft extends Azure management to the private cloud with Azure Arc appeared first on Inside Track Blog.

Now MSX Insights provides a single source of truth to more than 40,000 users, including salespeople, their managers, leaders, and multiple operations and finance teams across Microsoft. Based on Microsoft Fabric, a suite of Microsoft Azure technologies that includes OneLake, Data Factory, Synapse, Azure Analysis Services, and Power BI, MSX Insights is a project of the Microsoft Commerce and Ecosystem team, which powers, transforms, and protects our organization.

“Once we began using Azure and then Power BI, the technology limitations that had been holding back our data unification were eliminated,” says one of the project sponsors, Michael Toomey, senior director of business operations and programs for Microsoft Worldwide Sales Engineering. “Was it finally possible to get to a single view of our commercial business that everyone could understand?”

That’s exactly what happened when Microsoft Customer and Partner Solutions (MCAPS), Microsoft Finance and Data Experiences, and Customer Experience Data Engineering (CX Data) collaborated to create today’s comprehensive Microsoft business intelligence platform: MSX Insights.

[See how we automated our legacy revenue processing systems with optical character recognition (OCR) technology. Find out how we reinvented sales processing and financial reporting with Azure.]

Reports and metrics that didn’t add up

At Microsoft, it’s critical that decision makers have access to data they can trust across the sales pipeline, contracts, revenue, and consumption. But when they pulled reports from separate systems or used data updated at different times, the numbers and results often didn’t match.

“We used to have a lot of complaints,” says RJ Smith, principal group engineering manager with Microsoft Commercial Business. “I talked to people in Paris, Munich, Sydney, and they said that those reports wouldn’t load or that they didn’t show the right data.”

Praveen Vittalrao Ambekar, a principal group program manager for CX Data, also analyzed the MSX customer experience to find out what was driving the support volume.

We needed a 360-degree view of our customer to correctly evaluate key metrics. We wanted end-to-end visibility.

– Michael Toomey, senior director of business operations and programs, Microsoft Worldwide Sales Engineering

“Multiple data platforms powered these reports,” Ambekar says. “The insights weren’t aligned across sellers, managers, and leaders, and that was causing a lot of churn for the team. The groups were looking at the data from different angles.”

Those weren’t the only problems. The scope of the available reports didn’t fulfill the needs of senior executives.

“We needed a 360-degree view of our customer to correctly evaluate key metrics,” Toomey says. “We wanted end-to-end visibility.”

Although we empower our teams extensively to develop their own reporting platforms, we were seeing broad duplications of effort and cost. “It was a highly federated budget model,” Toomey says.

It was also risky. As people developed one-off solutions using copies of datasets, it became harder to secure the information and enforce compliance with standard data-handling practices.

“The more replication you have, the less likelihood that everyone’s compliant with the rules,” Ambekar says.

The insular systems also impacted engagement and satisfaction levels for our partners and customers.

“Close coordination across sales teams, partners, marketing, and operations is critical for our customers to get a connected experience,” Toomey says. “It’s impossible to achieve that if we have multiple datasets with mismatched data on opportunities, consumption, licenses, revenue, and other kinds of information.”

Multiple waves of data handling improvements

After we began using a standardized system, teams could migrate from competing products and use the same software regardless of department. The solution has stood the test of time.

“Power BI as a product is almost 12 years old and so is our platform,” Toomey says. “It has successfully adapted through the transitions of Microsoft’s core business model and the priorities of multiple engineering leaders—and our commercial business as a whole. We always need a central place to go and get insights.”

Then we launched Microsoft Azure cloud computing services, making it easier for users in different departments to access the same source of data.

“We started to take the approach of giving people what they needed based on their roles,” Toomey says.

That might be a seller who wants to see their scorecard broken down by account, a manager who needs an aggregate of the entire team’s pipeline, or a leader looking for patterns and trends over time. Microsoft Azure was a major enabler for this new direction, and Microsoft Power BI was the team’s choice of a front end for the evolving business intelligence platform.

“We had a quarterly business connection, an event where we bring all the executives together, area by area, segment by segment,” Toomey says. “Several of us got together and worked for six weeks to automate the data handling. We moved everything into Power BI and ran visuals there.”

That proof of concept was a success, so the next step was to make a cultural shift to get to an aligned environment. To that end, the team built a community around Microsoft Power BI practitioners in the field. This BI round table community gets together a few times a month to share best practices, what they’re doing locally, and what has the potential to scale up.

“We tried to connect with the people building the tools and explain that this was a better way for them to be successful,” Toomey says.

The team also focused on increasing the tool’s speed for users around the world.

“It’s not a problem anymore,” Smith says. “In fact, performance metrics have improved 50 percent. We spent a lot of time on performance to make sure the JavaScript implementation in the browser works well.”

In November 2020, representatives from the teams who were most involved in creating MSX Insights came together to address one remaining issue: getting the data right.

“It was a result of partnership and alignment between the three different teams—MCAPS, Finance and Data Experiences, and Microsoft Commerce and Ecosystems,” says Diego Ulloa, a data strategy lead with Microsoft Worldwide Enablement and Operations who works on MSX Insights. “Together we consolidated data, set business rules, and designed the architecture.”

Power BI as a self-service tool has enabled more consumption of the insights. You don’t need layers of people to pull it into Excel.

– Praveen Vittalrao Ambekar, principal group program manager, Microsoft Partner and Sales Experience Business Insights

“We had to make hard decisions,” Ambekar says. “We had to align to one or the other’s hierarchy.”

The team divided the rules and definitions up by functional area. In the first six months after the launch of MSX Insights, we eliminated 80 percent of the user complaints associated with data hygiene and report accuracy. We’re also improving its ease of use over time.

“Power BI as a self-service tool has enabled more consumption of the insights,” Ambekar says. “You don’t need layers of people to pull it into Excel.”

Collaboration continues to improve data quality and integrity

The group that built MSX Insights isn’t done yet. There’s a robust roadmap planned for the coming months and years.

More trends and customer reports are on the agenda. MSX Insights and the Partner Sales Experience are still evolving, and several other teams are now contributing to these platforms.

“We’ll continue to evolve the user experience,” Ambekar says. “We want the user interface to really match how and where people are working, embedding insights more directly into the experience.”

The collaboration across teams is helping avoid the duplication of efforts.

“Because we hold each other accountable when we go through and talk about the designs, we’re doing it once and doing it well,” Smith says. “It’s better by virtue of us working on it together.”

The path ahead for MSX Insights includes continuous rollouts of additional services and functionality using the latest capabilities in Microsoft Fabric and Power BI. We’re also looking at ways to integrate AI throughout the experience in an effort to enable faster decision making.

“By introducing these abilities to partners, we’re allowing more teams to create a comprehensive set of reports,” Ulloa says. “We’re taking our narrow vision and extending it to a One Microsoft model.”

• Unpack creating a modern data governance strategy to accelerate digital transformation at Microsoft.
• Discover how citizen developers use Microsoft Power Apps to build an intelligent launch assistant.
• Explore providing modern data transfer and storage service at Microsoft with Microsoft Azure.

Share your feedback with us—take our survey and let us know what kind of content is most useful to you.

The post Revamped Microsoft business intelligence platform boosts data handling and builds trust appeared first on Inside Track Blog.