{"id":4924,"date":"2019-11-07T10:35:23","date_gmt":"2019-11-07T18:35:23","guid":{"rendered":"https:\/\/www.microsoft.com\/insidetrack\/blog\/?p=4924"},"modified":"2023-06-08T12:56:45","modified_gmt":"2023-06-08T19:56:45","slug":"unexpected-cloud-cost-spike-spurs-optimization-movement-inside-microsoft","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/insidetrack\/blog\/unexpected-cloud-cost-spike-spurs-optimization-movement-inside-microsoft\/","title":{"rendered":"Unexpected cloud cost spike spurs optimization movement inside Microsoft"},"content":{"rendered":"
This content has been archived, and while it was correct at time of publication, it may no longer be accurate or reflect the current situation at Microsoft.<\/p>\n<\/div>\n<\/div>\n
When Microsoft moved its internal workloads to Azure, operational costs shot up\u2014something the team had to find a way to fix quickly.<\/p>\n
The transition from physical datacenters to virtual cloud computing happens in waves. Microsoft\u2019s journey began five years ago when the company\u2019s Microsoft Digital organization deployed many of its internal computing resources to the cloud using Microsoft Azure<\/a>.<\/p>\n The results were unexpected. The cost of doing business shot up, surprising employees like Paul Rojas, a program manager who now works on Microsoft Digital\u2019s Azure Optimization Team, which didn\u2019t exist at the time.<\/p>\n \u201cIf we were just doing a lift-and-shift, why were we seeing an increase in spending?\u201d Rojas says. \u201cWe realized that if we were seeing these issues, our customers would be having them as well.\u201d<\/p>\n The challenge with cloud computing is that once you flip the switch on, the meter keeps running until you switch it off. At Microsoft, which has a legacy of building its own always-on datacenters, employees were used to provisioning a server in a datacenter and forgetting about it.<\/p>\n \u201cThat can make Azure expensive,\u201d Rojas says. \u201cThat was something we weren\u2019t ready for.\u201d<\/p>\n Microsoft employees were billed for their Azure use just like customers, so optimization became an external and internal priority. Optimization starts with building awareness of spending and usage down to the resource level. To do this, teams needed tools for metering and tracking current usage, but they also needed to change their mindset and prioritize workflow optimization.<\/p>\n It would take a few stops and starts before the powers-that-be inside of Microsoft realized that they\u2019d need to build tools to track spending and find opportunities for cost optimization.<\/p>\n [Find out Microsoft is reducing its carbon footprint by tracking internal Microsoft Azure usage<\/a>. <\/em>Learn how to use Azure Cost Management to monitor Azure spending and optimize resource use<\/a>.]<\/em><\/p>\n Optimizing for efficiency and cost savings<\/strong><\/p>\n The Azure product group called on Microsoft Digital, which manages all of Microsoft\u2019s internal workloads, to carry the flag of Azure optimization for Microsoft employees and customers. This led to the creation of the Azure Optimization Team. Their charter? Enable modern and cost-optimized cloud platforms for Microsoft Digital and Microsoft.<\/p>\n \u201cWhen you design or build something, you want to make sure you\u2019re not using a semi-truck to accomplish a task when you could use a small car,\u201d Rojas says. \u201cThe same principle applies to your compute and storage resources in Azure.\u201d<\/p>\n One of Rojas\u2019s successful partnerships has been with Deepak Agrawal, a senior program manager who works on Azure Data Explorer<\/a> (ADX), a cloud service for storing and running interactive analytics. Agrawal\u2019s team responds to support tickets from Azure customers about optimization, and working with Rojas\u2019s team presented the opportunity to provide recommendations on an even larger scale.<\/p>\n \u201cThis was our first effort to go from a reactive engagement to a proactive engagement,\u201d Agrawal said.<\/p>\n The partnership began when Agrawal presented his strategy for Azure optimization at a cloud optimization conference.<\/p>\n \u201cADX is in the top three consumers of computing storage,\u201d Agrawal says. \u201cIt was paramount for us to go in front of customers to talk about how they could also save money.\u201d<\/p>\n Rojas also attended this conference and saw an opportunity for a partnership to optimize his own team\u2019s use of ADX and share these recommendations to all customers.<\/p>\n \u201cIf you can widen the scope and provide recommendations to all Azure teams using ADX, that would have a bigger impact,\u201d Agrawal says. \u201cPaul wanted to bring insights from the conference into a more consumable stream that we could share externally.\u201d<\/p>\n Currently, Agrawal\u2019s team is partnering with Rojas\u2019s team to generate Azure optimization recommendations for teams in Microsoft Digital based on current usage.<\/p>\n \u201cWe hope to provide actionable recommendations based on customer usage of the ADX platform,\u201d Agrawal says.<\/p>\n Developing a partnership<\/strong><\/p>\n When developing relationships with teams like ADX, Rojas starts by understanding a team\u2019s goals.<\/p>\n \u201cWe want to create the understanding that we are there to help them achieve their goals or navigate obstacles they may have had in the past,\u201d Rojas says. \u201cOur goal is to have a symbiotic relationship.\u201d<\/p>\n Then, Rojas\u2019s team has a conversation with team members to highlight their wins in past projects and make a plan for the future. To come up with personalized recommendations for the team, Rojas requests access to a team\u2019s data, which is used to measure everything from utilization to overall system health.<\/p>\n \u201cThe ADX team was more than happy to give us the data, and they wanted us to find instances of idle subscriptions that hadn\u2019t been used,\u201d Rojas says.<\/p>\n From there, developers on Rojas\u2019s team used the data they were given to develop recommendations for adjusting the size of their instances or deleting unused software. Rojas\u2019s team found that internal teams were spending $250,000 a month on idle subscriptions. The release of this recommendation led to $10,000 in cost savings. Rojas says that this could lead to hundreds of thousands of dollars in savings.<\/p>\n \u201cWe know that making users aware of optimization opportunities creates less demand on the system,\u201d Rojas says. The money that\u2019s saved \u201ccan be used for other avenues like training, headcount, or investment in high-priority items.\u201d<\/p>\n Beyond cost savings, optimization recommendations also empower engineers to make informed decisions about how they use Azure based on usage and spending data. After creating optimization recommendations, Rojas\u2019s team consulted with the ADX team to see if their suggestions for Azure optimization aligned with the team\u2019s needs and expectations.<\/p>\n The long-term goal is to create a production-grade optimization recommendation engine that surfaces specific recommendations for optimizing Azure use. Agrawal hopes to share more than 20 recommendations for cost savings, best practices, and performance improvement on Azure Advisor<\/a> and internally to Microsoft employees.<\/p>\n \u201cIt\u2019s valuable to provide best practices in Azure Advisor by looking at the queries that have been executed on the data and provide specific recommendations,\u201d Agrawal says. \u201cThis enables customers to better manage their configurations and optimize their use of the platform.\u201d<\/p>\n \u201cIt\u2019s a new exercise for us, and leveraging this platform, experience, and knowledge is a great way to go forward and still provide a good experience for customers,\u201d Agrawal says.<\/p>\n