Microsoft is using Kanban to drive engineering improvement and streamline workflows at Microsoft.

In its simplest form, Kanban involves creating a set of cards that track manufacturing or other step-by-step processes. These cards, tacked to a corkboard, can be used to highlight trouble spots and avoid overcapacity. That latter quality helps Kanban users resist loading up a job with too many side tasks.

“I learned about Kanban when I was in the Marine Corps,” says Ronald Klemz, a senior software engineer manager for Microsoft Commerce and Ecosystems. “When I joined Microsoft, I could see how it applied to software engineering.”

As it turns out, Microsoft already had an internal Kanban evangelist: Eric Brechner, who has since started his own company, leaving behind an influential legacy and a must-read book.

[Learn how Microsoft uses Azure Resource Manager for efficient cloud management.]

Although Kanban at Microsoft had a toehold, most engineers still used “scrum” or “Waterfall” development frameworks. Both attempt to help teams manage and assign workloads. Scrums, for instance, consist of regular planning meetings followed by two week to month long sprints that are meant to complete a particular stage of work.

We had a need to really visualize our work, which scrums couldn’t provide. Another engineer said, ‘Hey, have you heard about Kanban?’ We did some research and decided this was a good fit.

—Jon Griffeth, software development engineer, Microsoft Commerce and Ecosystems

While plenty of good work has come out of scrums and Waterfalls, they are not always ideal for driving engineering improvement. In scrums, for instance, the regular meetings can be time consuming and even though scrums are designed to break big jobs into manageable pieces, teams can still become overwhelmed if customers add new requirements on the fly.

“At the start of each two-week scrum cycle, you’re expected to know everything that you’re going to do in those two weeks,” says Snigdha Bora, an engineering lead with Microsoft Digital, the organization that powers, protects, and transforms Microsoft. “But there are things that will happen in those two weeks that you can’t know in advance. All of that goes away with Kanban because it has no limitations or artificial boundaries of a week or two weeks.”

“We were having problems managing with scrums, and were constantly missing sprint conclusions,” says Jon Griffeth, a software development engineer and program manager for Microsoft Commerce and Ecosystems. “We had a need to really visualize our work, which scrums couldn’t provide. Another engineer said, ‘Hey, have you heard about Kanban?’ We did some research and decided this was a good fit.”

Whether built with simple paper tags or using more sophisticated software versions, a Kanban board shows rows of cards arranged in columns that represent stages of a project’s workflow. Each card contains a specific task and who is responsible for it.

One of Kanban’s most valuable aspects is that each column is designed to self-limit work in progress. If an extra card is added that exceeds the agreed upon limit of tasks, the column heading might light up red, indicating a possible bottleneck that could delay work.

“It helps to simplify the workflow, so people aren’t getting hit with all kinds of sudden, ad hoc projects,” Klemz says. “They’re able to focus on the agreed-upon workflow.”

Griffeth agrees.

“When we would want to add an item to the workflow, Kanban helped us have more objective conversations about what we could and couldn’t do,” Griffeth says. “It also brings accountability within the team, and people get to pick a task and run with it. Then, if they are done with it, they can go to the next item on the priority list.”

If you finish a model, you don’t have to go to the project manager and ask what needs to be done next. You can see what’s next right on the Kanban board, pick up the next step and run with it.

—Baala Arumugam, senior software engineer, Microsoft Commerce and Ecosystems

That last point underscores another advantage of how Kanban at Microsoft drives engineering improvement: Its visual nature makes it easy for someone who is a newcomer to a team, has been on vacation, or is a part-timer, to look at the Kanban board and immediately see what needs to be done.

“With Kanban, it’s much easier to pick things up if you’ve been gone for a couple of days or if you’re just coming into the team,” says Baala Arumugam, a senior software engineer for Microsoft Commerce and Ecosystems. “And if you finish a model, you don’t have to go to the project manager and ask what needs to be done next. You can see what’s next right on the Kanban board, pick up the next step and run with it.”

That is especially handy in a time when COVID-19 has essentially all Microsoft engineers working remotely, often in different time zones. With Kanban boards, often created with Microsoft Azure DevOps, they can always immediately see the status of a project.

Binu Surendranath’s team owns the tools, processes, and controls to ensure that Microsoft’s preferred suppliers and partners are paid in a timely way once invoices are approved. They also ensure tax and other statutory compliances globally, provide tax and statutory compliance information, and report payments to the Internal Revenue Service.

Those multiple workflows led to siloed work, with different members of the team unaware of what co-workers were working on, or how their work had an impact on others.

“Everybody had their own priorities,” Surendranath says. “If I’ve finished one part of the puzzle, I celebrate a victory. But that didn’t really make a dent in the overall project. We support global businesses that are expanding exponentially. Having common, quantifiable business outcomes for everyone to work towards became an obvious need.”

Kanban has helped his team create a more collaborative work environment while still giving engineers plenty of freedom for innovation and simplification to positively impact customer experience and business needs, Surendranath says.

Sounds good. But what about concrete benefits to Kanban at Microsoft? There are plenty.

“Gone are the days when we’d spend nine months on a quarterly update,” Surendranath says. “Now when you close and open Outlook, you have a new Outlook because of the frequent updates Microsoft makes to it and other apps. That takes a more agile development approach that Kanban works well with.”

The agility plays well with Microsoft customers, who like to see product improvements that are rapid and seamless. The same goes for the business expansion of Microsoft Azure and data center launches and announcements.

“From the time Microsoft CEO Satya Nadella announces a roll-out, we have just a few weeks to get everything up and running,” says Surendranath. “Kanban has really enabled us to meet that need with a high level of confidence and transparency. Kanban dashboard enabled real-time transparency on progress of business priorities and allowed us to manage our OKR (Objectives and Key Results) closely and were able to drive our monthly business reviews more efficiently. We started bringing up the dashboard during our business reviews to give transparency to all global stakeholders, which eventually helped build stronger trust.”

Kanban also helps Microsoft teams more effectively manage and deploy global statutory laws and compliance, which can change rapidly with predefined timelines and in most cases are non negotiable.

Griffeth’s engineers, meanwhile, were assigned the task of creating a new purchase order workflow for a team in India.

“We tracked a lot of what had to be done in Kanban,” he says. “It helped us see where a bottleneck might be, such as the product owner flooding the first step of the process with a lot of requests, or if code validation becomes a problem.”

The result: A smoother process, happier customers, and a team that worked well together. The team also saw improved productivity because no one was spending time in scrum meetings or working as scrum master. Internal customers and business groups embraced real-time transparency, accountability, and predictability on engineering dependencies.

Kanban continues to be a learning process for Microsoft engineers using it, and it has not yet gained truly widespread acceptance. But it has shown a path to make software development faster and more trouble-free, while helping teams work together more effectively.

Learn how Microsoft uses Azure Resource Manager for efficient cloud management.

Share your feedback with us—take our survey and let us know what kind of content is most useful to you.

The post Deploying Kanban at Microsoft leads to engineering excellence appeared first on Inside Track Blog.

Adopting rigorous design standards is helping Microsoft get better at something very important to the company—getting accessibility right inside its own walls.

Microsoft’s journey to transform its approach to accessibility started when Microsoft CEO Satya Nadella took the helm in 2014, says Tricia Fejfar, partner director of user experience in Microsoft Digital, the organization that powers, protects, and transforms Microsoft. Nadella sharpened the company’s focus on accessibility in 2017, when he penned a moving essay describing his experience raising a child with cerebral palsy.

“That really got us thinking about accessibility internally,” Fejfar says. “Employees are more productive and engaged when they have simple, easy-to-use tools, and accessibility is a very important part of that DNA.”

More than 1 billion people on the planet identify as having some form of a disability, so building experiences that are accessible to all Microsoft employees makes a difference every day.

“Being able to do my job at Microsoft based on my skills and not be blocked by my blindness has made a big difference in my life,” says Manish Agrawal, a senior program manager for the Accessibility team within Microsoft Digital.

Agrawal, who is blind, works to make Microsoft products more accessible to people with disabilities. It’s about creating an inclusive work environment where everyone can succeed.

“For me, it’s not just about making products accessible for Microsoft employees to help them get their work done,” he says. “It’s also about supporting employees with disabilities and ensuring that Microsoft builds a diverse and inclusive workforce across the spectrum of abilities.”

Fejfar adds, “Designing for and building experiences that reflect the diversity of the people who use them makes sure we put our people at the center of our work. Until people recognize that, and honor it in the work they do, they can’t begin to make sure what they build will take care of everyone’s needs.”

It’s about understanding why you build something and who will use it. Microsoft calls it being human-centric and customer obsessed.

“Building accessible experiences is not a compliance effort or a checklist of guidelines,” Fejfar says. “It’s about thinking of the user at all stages of the development process so you build usable, delightful, and cohesive end-to-end experiences.”

Hiring and supporting people with disabilities makes good sense for the company and helps attract top talent.

“Millennials choose employers who reflect their values, and diversity and inclusion are at the top of their list,” Fejfar says. “They make up 75 percent of the global workforce.”

Making a difference in the lives of people like Agrawal is what brings people to the Accessibility team, Fejfar says. “We’re here because we want to make sure the internal products that our employees use every day are accessible,” she says.

[Find out how building inclusive, accessible experiences at Microsoft is a catalyst for digital transformation. Learn how Microsoft enables remote work for its employees.]

Adopting a coherent design system

Nadella sharing his story led to a company-wide pivot toward accessibility and improving employability for people with disabilities at Microsoft. One of the initiatives connected to this goal was creating a set of coherence design standards that teams can use each time they builds new tools and services for employees.

“Using a coherent design language reduces engineering costs while increasing engineering efficiency,” Fejfar says. “That makes what we build predictable to our users, which increases engagement and builds trust.”

Microsoft Digital’s design system is built on top of Fluent, Microsoft’s externally facing design language, which makes it feel more like Microsoft.

“Building coherently means something very specific to us,” Fejfar says. “It means designing and coding accessible and reusable UI components, interaction patterns, brand, and other guidelines to build predictable experiences for our employees.”

These design standards have allowed Microsoft to not only consider accessibility as part of every internal project. They also consider accessibility at every step along the way, from idea, to construction, to release. That makes its products accessible to as wide a range of people as possible, which creates new opportunities and better experiences for everyone who works at Microsoft.

Accessible design benefits everyone

Agrawal cites closed captioning as an example of a widely useful accessibility tool that is now used for far more than helping people with hearing impairments watch TV or follow a presentation. Creative uses of the capability include helping audiences understand someone with a heavy accent, following along on TVs placed in loud environments like airports and bars, or allowing someone to watch TV while their partner sleeps.

In fact, closed captions or subtitles are so popular with the general population that game maker Ubisoft reported that more than 95 percent of the people who play their popular Assassin’s Creed Odyssey game keep subtitles turned on. “When you build for accessibility, you end up building a much more compelling product,” Agrawal says.

Moreover, it’s simply good business sense to ensure that talented people such as Agrawal are empowered to make a significant contribution to companies such as Microsoft.

“We need to make sure all the applications and experiences that we build empower everyone who works here to not only do their work, but to have full, rich experiences while they’re at work,” Fejfar says. “Without accessible tools, people can’t do their best work, and if people can’t do their best work, our company, our culture, and our customers are directly impacted.”

For a transcript, please view the video on YouTube: https://www.youtube.com/watch?v=XhN1tnBcYLo, select the “More actions” button (three dots icon) below the video, and then select “Show transcript.”

Designing new employee experiences

One telling example of Microsoft Digital’s coherent design approach to accessibility is Microsoft MyHub, a new one-stop shop for employees to get their “at work” stuff done at work, like getting worksite access, taking time off, checking stock rewards, and finding out what holidays are upcoming.

It was also vital to make sure the app experience would be fully accessible, says Bing Zhu, principal design manager in Microsoft Digital’s Studio UX team.

“Before we built the app, our employees had to deal with as many as five to eight different tools almost every day,” Zhu says. “Each experience was different than the last one, and not all of them were as accessible as we needed them to be.”

This fragmented experience was difficult for everyone to navigate and very hard to keep accessible for people with disabilities.

“We used our coherent design system to build a unified, consistent, and accessible experience for our employees,” Zhu says. “Using that as our guide, we were able to design an application that all Microsoft employees can use.”

Not only is Microsoft MyHub compliant with Web Content Accessibility Guidelines (WCAG), but it also received a strong usability grade by employees with a spectrum of vision disabilities.

Crucially, the new app was built with accessibility in mind at every stage of its development cycle, Agrawal says.

“We reviewed the design for every feature for accessibility and beta tested the app’s accessibility every time a new feature was implemented,” he says. “We made sure it was accessible for all of our users at each step in the development process.”

One example of how the team that built Microsoft MyHub was guided by Microsoft Digital’s coherence design system was in how it made every interaction and visual element accessible.

“Our coherence design system—which is an extension of Microsoft’s Fluent design system—alongside the accessibility guidance that we provide, helped the MyHub team start incorporating accessibility into their app from the get-go,” says Anna Zaremba, a senior designer on Microsoft Digital’s Coherence team. “Our coherence design system provides components with built-in accessibility that Microsoft Digital’s product teams, like the team that built MyHub, use to create their experiences.”

Work that makes a difference

It’s striking to hear employees in Microsoft Digital talk about the deep satisfaction they take from making products more accessible.

“The greatest reward is hearing from people who have benefitted from our work,” Zaremba says. “I really like the fact that we are doing work that helps the entire company and drives a greater awareness of accessibility.”

Though Microsoft is among the companies pushing hard to build accessibility into everything it does, there is still much work to do. One in 10 people who identify as having some form of disability don’t have the assistive technology they need to fully participate in work and society.

Going forward, Microsoft Digital will continue designing with accessibility as a top priority, using the developmental model it uses to build solutions like Microsoft MyHub as a template for creating the company’s next generation of employee tools.

“We’re still learning this process ourselves,” Zhu says. “We’re figuring out how to make accessibility and design work with program managers and engineers to create even more opportunities for access. It’s an exciting challenge.”

And one that will open doors for Microsoft employees—and others.

“I really love building software anyway,” Agrawal says. “But it’s great to be part of a team that is working to make Microsoft a more inclusive place to work. It has a real impact on people’s lives.”

• Learn more about creating an accessibility training program for your team.
• Dive into the latest accessibility fundamentals training course on Microsoft Learn.
• October is National Disability Employment Awareness Month 2020—learn more about the 75th anniversary of this event in the US here.
• Read how Microsoft is sharing how many of its employees identify as having some form of disability.
• Read how people with disabilities shape technology people use every day.
• Find out how building inclusive, accessible experiences at Microsoft is a catalyst for digital transformation.
• Learn how Microsoft enables remote work for its employees.

The post Microsoft’s fresh approach to accessibility powered by inclusive design appeared first on Inside Track Blog.

At Microsoft, we pride ourselves on working fairly and honestly with the people we do business with. The increasing use of online commerce, however, has made trade screening more complicated and we’ve had to work hard to maintain compliance and increase efficiency.

—Joseph Hindo, principal software engineering manager, Microsoft Digital

Microsoft Trade and Corporate, External, and Legal Affairs (CELA) teams, in partnership with Microsoft Digital and SAP, worked together to deploy a reliable and scalable compliance management solution that combines the rich feature sets of SAP Business Integrity Screening (BIS), the processing power of S/4 HANA databases, and the scalability and agility of Microsoft Azure.

SAP BIS trade screening is one aspect of the company’s larger Trade Screening Transformation (TST) program, which ensures Microsoft compliance with embargoes, sanctions, and denied party regulations. As part of the transformation, SAP BIS helps Microsoft trade screen with more speed and efficiency.

“At Microsoft, we pride ourselves on working fairly and honestly with the people we do business with,” says Joseph Hindo, a principal software engineering manager with Microsoft Digital, the organization that powers, protects, and transforms Microsoft. “The increasing use of online commerce, however, has made trade screening more complicated and we’ve had to work hard to maintain compliance and increase efficiency.”

To combat that, the company has launched its new Microsoft Azure-based screening tool.

How does it work?

As part of Microsoft’s TST program, the tool screens all parties that seek to do business with Microsoft. It does so quickly and accurately despite having to sift through massive amounts of data in ways that comply with laws and regulations that vary by country and region.

We wanted to have a system so that when an account is created and the user clicks ‘save,’ the customer is screened in real time. In half a second, we now can screen a customer’s name and other details and provide an immediate response as to whether or not business with Microsoft is allowed.

—Antti Lamberg, senior program manager, Microsoft Digital

With Microsoft Azure and the HANA database, SAP BIS Trade Screening runs under the hood of several sales channels, flagging potential problems within milliseconds of a customer interaction taking place, and reducing potential business bottlenecks. Not only does TST help Microsoft solve a major business challenge and meet regulatory requirements, its deployment of SAP BIS S/4 HANA might soon be the largest in the world to date.

[Learn how Microsoft examines SAP transactions with Azure Anomaly Detector. Find out how Microsoft optimizes SAP for Azure.]

A fast deployment

Building a new trade screening tool happened in just five months, from the first proof of concept meetings in November 2019, to a go-live in April 2020.

Microsoft has long worked to enforce fair trade practices. For several years, it has used multiple trade screening systems to look for questionable transactions or entities. But each had its own gaps and challenges. The TST system gives Microsoft the opportunity to improve customer experiences through faster detection, using the power of predictive decision-making and integration with other SAP and non-SAP systems, which reduces the time to market for global expansion and creating new customer records.

“We wanted to have a system so that when an account is created and the user clicks ‘save,’ the customer is screened in real time,” says Antti Lamberg, a senior program manager with Microsoft Digital. “In half a second, we now can screen a customer’s name and other details and provide an immediate response as to whether or not business with Microsoft is allowed.”

Now, Microsoft screens companies and individuals with whom the company does business at the time of record creation or updates, no longer needing to screen a customer’s multiple separate transactions in a given day for a sales channel.

It’s really, really fast

What is perhaps most impressive about the SAP BIS Trade Screening tool is that it’s blazing fast—as Lamberg notes, it needs less than 500 milliseconds to identify potential trade blocks. And that can involve some millions of records processing 24/7/365.

“That was a tough one,” says Jasmit Kohli, a senior software engineer and SAP expert with Microsoft. “That would be four-nine performance, meaning 99.99 percent delivery against a target metric. We had to think outside of posting on a single system. We’re implementing an active-active node structure so if one node is down, it can route the call to a second node. That way, the availability is always there.”

Azure is a beautiful product in terms of building solutions at scale. It offered the services and robust infrastructure that allowed us to build a high-performance system. It was fascinating to leverage the standard Azure service offerings to meet different business use-cases.

—Jasmit Kohli, senior software engineer, Microsoft Digital

Steps engineers took to reach this goal included determining the cost of every millisecond saved, finding opportunities without compromising security or compliance. They applied best practices for caching, concurrent processing, and other aspects to get consistently fast results. And they ensured that each component of SAP BIS Trade Screening had rich telemetry for good insight into how the tool is performing.

Microsoft Azure was an important tool for making the SAP S/4 HANA database work well.

“Azure is a beautiful product in terms of building solutions at scale,” Kohli says. “It offered the services and robust infrastructure that allowed us to build a high-performance system. It was fascinating to leverage the standard Azure service offerings to meet different business use-cases.”

A big team effort

The team working on SAP BIS trade screening had several challenges, not least of which was the tight deadline. Another big task was to connect SAP S/4 HANA to Microsoft Azure and integrate it into Microsoft sales workflows without excessive customization. The third was a challenge now familiar to nearly everyone in COVID backdrop: working as a remote and distributed team with several global development teams, including SAP in Germany and Microsoft teams in the US and India.

“We talked and brainstormed quite a bit,” Hindo says. “We flew to Germany to meet with SAP and learn more from them about their capabilities, and how those would fit into how we would screen business data.”

All told, some 30 teams worked on the SAP BIS Trade Screening tool, with more than 100 people working on the engineering, and many others on the business side.

“I really enjoyed the way the team worked together,” Hindo says. “It was a great collaborative effort between the teams around the world to get this thing across the finish line.”

With the basic architecture of SAP BIS Trade Screening now in place, Lamberg says the engineering work on the product has focused on future-proofing the technology. In addition to unifying data today, the trade screening tool is designed to accept additional AI capabilities, integrate with future Microsoft Azure components, and work with language detection. For the latter, the SAP BIS Trade Screening tool could identify different languages and send the system’s report to a specialist in that language.

“We upgraded a new version of the tool in February, and we upgraded S/4 HANA at the same time,” Lamberg says. “And then we heard from SAP that we were the first company worldwide to actually do that. We were happy about that.”

To ensure Microsoft Runs on Trust, Microsoft’s Azure-based SAP BIS trade screening tool is a powerful component of the trade screening compliance program.

• Learn how Microsoft examines SAP transactions with Azure Anomaly Detector.
• Find out how Microsoft optimizes SAP for Azure.

The post New trade screening tool boosts Microsoft appeared first on Inside Track Blog.

Finding rogue access points on Microsoft’s network is an important mission for our IT teams.

Networked devices have come to dominate the IT world, and their prevalence has led to more complex and vulnerable gateways. As a result, employees within Microsoft and many other large organizations regularly bring in their own wireless devices. Using a wireless router designed for home office use or a wireless speaker system might seem harmless, but these rogue access points (APs) pose serious security risks.

An unauthorized user could be sitting in the parking lot and you just knowingly or unknowingly gave them access to the corporate network.

– Pete Fortman, principal engineer, Microsoft

In the case of a wireless router designed for home use, it might have a default password that’s literally “password” or the device’s brand name. That could give drive-by hackers easy access to an enterprise’s network.

“An unauthorized user could be sitting in the parking lot and you just knowingly or unknowingly gave them access to the corporate network,” says Pete Fortman, a principal engineer for Microsoft who focuses on security.

With networking built into more and more devices, an increasing number of seemingly benign APs can also act as connectors. That means that in spite of strict segmentation within our overall network environment, threats can piggy-back on increasing numbers of rogue APs to gain access to corporate networks.

Eliminating these vulnerabilities is essential to maintaining a Zero Trust environment.

The danger of rogue APs

Once inside, bad actors can wreak havoc. They can steal intellectual property, flood a network with useless data, or set up conversations between people who think they’re speaking with each other when in fact they’re talking to the attacker.

One of the most damaging outcomes is a ransomware attack. That’s a type of malware that blocks access to critical data or systems until the target pays a ransom, and they can be massively disruptive in terms of both operations and customer trust.

Beyond that, rogue APs can interfere with legitimate wireless traffic—often by simply competing for airtime with the unwanted device. “It’s like a conference room with 18 seats, but 50 people are in the room and they’re all trying to stream something wirelessly,” Fortman says.

We’ve fought to keep rogue APs off our network for years. But as devices become more complex and plentiful, they’ve also become more difficult to detect. That doesn’t just increase the number of risky APs attached to our network. It also vastly increases the amount of telemetry that IT teams have to address, resulting in greater data volume and complexity.

To combat that, we’re applying machine learning and other advanced techniques to track rogue APs down.

When we began examining additional telemetry to find rogue access points in 2019, Fortman was surprised by what we uncovered.

“We had rogue devices all over the place,” Fortman says. “We kept the data private for a while to prevent adversaries from knowing what we can and cannot detect. When we shared the data more broadly, there was a collective gasp as people realized what was going on.”

[Learn how Microsoft 365 helps create a secure, modern workplace. Find out how Microsoft ensures security with Windows Hello for Business.]

Tracking down rogues

Obviously, rogue AP vulnerabilities aren’t good at a company that relies on Zero Trust to ensure security.

Gathering all this information into one place was a feat unto itself. We had to do it twice for two different data sets. Then we had to correlate the data sets together, and then look at suppression technology.

—Vincent Bersagol, senior software engineer, Microsoft

An engineering team within Microsoft Digital Employee Experience (MDEE), the organization that powers, protects, and transforms our internal technology, took on the challenge of identifying and removing rogue devices.

Finding rogue APs posed a substantial engineering challenge. Potentially thousands of devices from a wide range of manufacturers might be on the loose in the corporate network—all using different wireless protocols.

“Gathering all this information into one place was a feat unto itself,” says Vincent Bersagol, a senior software engineer for Microsoft. “And we had to do it twice for two different data sets. Then we had to correlate the data sets together, and then look at suppression technology.”

Microsoft’s data tools, such as Microsoft Power BI, Microsoft Azure Data Lake, and Microsoft Azure Synapse, played a key role in collecting and correlating the data. “That was a great way to visualize all this data for folks to have a look at it,” Bersagol says.

Our expertise in machine learning also proved helpful for finding rogue APs. We used it to sort through the correlations between wired and wireless devices.

“We used a clustering algorithm that allowed us to tease out all the media access control (MAC) addresses that were statistically related to each other in a way that humans couldn’t see,” Bersagol says.

Many access points have commonly identifiable designs we can determine by looking at multiple sets of network telemetry, including the MAC addresses. Finding these identifiable designs began with a manual examination of the rogue APs we’d already discovered. We recognized that requiring a sample of every type of rogue AP to generate a manual identification to find new patterns would present problems as the project scaled.

But collecting all the wired and wireless telemetry to hunt for new rogue AP designs wasn’t enough. “That’s too much data for humans to sift through,” Bersagol says.

Instead, we ran a script that matched the two telemetry sets across all machines encountered. If the script found any correlated wireless and wired data, the odds were very high that they came from the same device—a rogue AP. We gained further confidence that we’d found a rogue AP when the correlated addresses came from within the same building.

So far, so good.

But some devices have designs that elude direct correlation using the existing telemetry. By using additional telemetry sources, we’ve been able to unearth devices that are more difficult to detect.

Still, even finding the simpler devices yields an impressive collection.

In the early stages of the project in October 2019, a sweep of about 100 buildings on the Microsoft campus unearthed more than 1,000 rogue APs.

COVID-19 plays a role (of course)

The COVID-19 pandemic had several impacts on the team tasked with finding rogue access points. Many rogue devices disappeared from the network because their owners were working from home.

The disruption also challenged some of the engineers working on the problem.

Blaze Kotsenburg, a software engineer, began work on the project in June 2020—his first month as a Microsoft employee. But onboarding, meeting new team members, and getting up to speed on the rogue AP project all took place over Microsoft Teams.

“I couldn’t go to my mentor Vincent and ask him for a 15-minute whiteboard,” Kotsenburg says. “I’d work on something for a few hours, then ping him and say, ‘Hey, I need some help.’”

In spite of these challenges, the entire team found new ways to collaborate and recreate the in-office dynamic. Diego Baccino, a principal software engineering manager, shares that the virtual work environment helped create a single team, rather than one team led by Fortman and one by Baccino.

“Working with two teams in parallel worked even better because of the remote situation,” Baccino says. “If I were to do this over again, I’d put even more emphasis on communication between everyone involved.”

This strong collaborative stance has remained as employees have transitioned from fully remote to hybrid work.

Pulling the plug

It’s possible to take a very fine-grained approach to finding rogue access points and booting them off a network, such as assigning traffic through their ports to a virtual local area network (VLAN), or by blocking the devices’ MAC addresses.

In this case, we opted for a more blanket approach: shutting down any port connected to a rogue AP. This technique proved simple and effective, and safer than trying gentler approaches.

There’s what Fortman calls “collateral damage” because when a port is shut down, its user might lose network connectivity for other devices in their office, and Microsoft loses visibility to anything connected to that port.

“Shutting down a port is a basic capability of wired access” Fortman says. “As more Zero Trust networking capabilities become available on the infrastructure, we’re leveraging them to proactively prevent some devices from connecting and to enact more precise rogue AP suppression through automated remediation.”

While our earlier work was about identifying, cataloging, and remediating accumulated rogue AP issues, we’ve now developed a more real-time approach. We’re using Azure EventHub and Data Explorer to handle real-time telemetry to help improve the security response time.

That set the stage for automated remediation. Now, when our systems detect a rogue AP, we can automatically suppress it through an automation platform that turns off the associated ports—no human intervention required.

Extending the lessons of rogue AP suppression

MDEE’s work tracking down and remediating rogue APs has been so successful that they’re preparing slices of that data to provide to Azure datacenter teams. They’ll use the lessons learned to enact their own rogue AP detection to fulfill regulatory requirements across different geographies throughout the world.

Finally, these capabilities are spawning other abilities across teams as well. MDEE is actively looking for opportunities to apply the platform they’ve created throughout Microsoft. That might eventually lead to a self-serve platform that other business groups within Microsoft can access for their own AP security needs.

As new threats emerge and old ones find new ways to cause problems, security is a constant challenge. At Microsoft, preventing unwanted intruders is a top priority, and digital sleuthing has helped us close off one more avenue that bad actors might use.

• Unpack implementing a Zero Trust security model at Microsoft.
• Discover running on VPN: How Microsoft is keeping its remote workforce connected.
• Explore how Microsoft ensures security with Windows Hello for Business.

The post Finding and remediating rogue access points on the Microsoft corporate network appeared first on Inside Track Blog.

Why?

Like many enterprises, Microsoft runs on SAP.

It uses the software to run everything from tracking servers in its supply chain to making sure the company’s 140,000 employees are paid on time. It has one of the largest SAP deployments in the world.

In fact, Microsoft manages 50 terabytes of SAP data (enough to hold nearly 7 million digital photos) on 700 Microsoft Azure virtual machines. The company’s SAP usage is doubling year over year, and it conducts 300 million operations per month.

In 2017, Microsoft Digital team orchestrated a massive lift-and-shift of the company’s SAP business process services, moving that data trove to 700 virtual machines.

The move was intended to save Microsoft operational costs, while also improving reliability.

And it succeeded.

Still, more could be done, says Sanoop Thrivikraman Nampoothiri, a senior software engineer in Microsoft Digital.

“We thought the benefits of moving to Azure could be even greater if we just took a few more steps,” he says. “So we designed a way to more efficiently manage the resources we use to power our SAP applications.”

The team had already made progress. Costs of managing the SAP workload dropped 18 percent during the first two years of Microsoft Azure operations. That was thanks to moving away from on-premises hardware and waterfall-based engineering practices and deploying upgrades. You can read more about that effort.

Moreover, lifting-and-shifting Microsoft’s SAP workload to Azure allowed the company to easily scale its SAP application to keep up with the explosive growth in usage.

But Microsoft Digital engineers sought further savings. They saw pain points such as the rising costs of running SAP on virtual machines, the company’s one-size-fits-all approach to server configuration, and the lack of an out-of-the-box way to dynamically scale SAP workloads.

[Learn how Microsoft monitors SAP end to end. See how Microsoft monitors end-to-end enterprise health with Microsoft Azure. Check out how Microsoft migrated critical financial systems to Microsoft Azure.]

Adapting SAP for the cloud era

One of the challenges with managing SAP in the cloud is that even though it now has more than 220 million cloud-based users, it’s not fully optimized for today’s elastic cloud infrastructure.

“SAP (in its current form) was designed back in the 1990s,” Nampoothiri says. “It’s an older architecture, and it’s not as modern as some of our other Azure services—especially web services. At the same time, it’s one of our busiest services, managing everything from finances to supply chains. And a lot of our customers are in the same position as we are.”

One result is that engineers tend to be cautious when managing mission-critical applications such as SAP, building in plenty of capacity to ensure customers always have access.

“When you design a system like that, you always design for peak load,” Nampoothiri says. “Most of our customers do the same. But Azure has a lot of flexibility that allows you to right-size systems.”

This leaves SAP application servers at a sweet spot for automation and optimization. Combining the oversight of Microsoft Azure Monitor with the power of Microsoft Azure Automation, these application servers can be scaled at will.

Microsoft Digital carefully monitors SAP usage and stability using Microsoft Azure Monitor and applies technologies such as predictive analytics to spot potential problems before they occur. That monitoring also measures the loads on SAP infrastructure, allowing engineers to clearly see usage patterns.

“The telemetry from Microsoft Azure Monitor helped us understand which workgroups have different loads,” says Karan Parseja, a Microsoft Digital software engineer in Hyderabad. “The next step was to build a solution that would decide which servers should run at lower load levels, and then automatically reduce the capacity for those servers. We also needed the solution to gracefully stop an application when needed.”

Enter auto-scaling, tight-sizing, and snoozing.

After the migration of 700-plus virtual machines (VMs) to Azure, we were constantly looking at the opportunities for further optimization of infrastructure resources.

– Santosh Rajput, senior software engineer in Microsoft Digital

Microsoft Azure runs SAP more efficiently with auto-scaling

The seed for auto-scaling SAP on Microsoft Azure came from a hackathon—an annual week-long event at Microsoft where everyone teams up with colleagues to work on ideas of their choice.

“After the migration of 700-plus VMs to Azure, we were constantly looking at the opportunities for further optimization of infrastructure resources,” says Santosh Rajput, a senior software engineer in Microsoft Digital. “During a hackathon, we came up with this idea of scaling in or out of SAP application servers automatically, in real time.”

Altogether, the team took three approaches to improve how Microsoft Azure runs SAP:

Auto-scaling. The team embraced an “infrastructure on demand” approach, in part because it’s easy to scale Microsoft Azure up as needed. Team members used the SAP Quick Sizer tool to estimate precisely how much VM capacity was needed, then scaled accordingly. And they shortened the planning horizon from several years to six months, enabling more precise adjustments to demand.

Tight-sizing. Most system demand peaks are predictable—quarter-end and year-end in particular. The Microsoft Digital team redesigned its VM array running SAP to correlate system capacity with anticipated peak demands.

Snoozing. Perhaps the biggest change was to move away from the always-on status of the original SAP setup. The Microsoft Azure team used Microsoft PowerShell to give the system the ability to “sleep” during quiet periods. But if someone is working on a weekend and needs access, the virtual machines rapidly come back online to do the work.

The reconfigured SAP/Microsoft Azure system also was redesigned with fewer points of failure and has a substantial degree of redundancy to guard against unexpected faults. It also has dual databases, which provide automatic failover in the event one crashes. That also makes it easier to perform system upgrades without interfering with work demands.

Still, perhaps the biggest task was finding a way to deploy these improvements in a way that allowed Microsoft’s SAP infrastructure to keep working smoothly while auto-scaling SAP on Microsoft Azure changes were made. Think of it as repairing a jetliner mid-flight—from the outside.

“We had to convince our stakeholders that this would really work without having an impact on the availability of the system,” Rajput says. “Any customer running SAP and Azure would have that concern as well.”

When we moved, we didn’t want to take any chances. We wanted to show people the best possible way to run SAP on Azure. So we were conservative and focused on availability for peak loads. But now we’re confident that Azure can handle SAP workloads, so now we’re working on optimization.

– Niranjan Maski, senior program manager in Microsoft Digital

Niranjan Maski agrees. He is a senior program manager for Microsoft Digital in Hyderabad.

“When we moved, we didn’t want to take any chances,” Maski says. “We wanted to show people the best possible way to run SAP on Azure. So, we were conservative and focused on availability for peak loads. But now we’re confident that Azure can handle SAP workloads, so now we’re working on optimization.”

Empowering customers to do more with SAP

Overall, auto-scaling SAP on Microsoft Azure reduced the cost of running SAP by another 18 percent and created a more robust system in the process.

Now used internally, these improvements may be rolled out for customers using Microsoft Azure and SAP. That would be an important stage in keeping Microsoft Azure abreast or ahead of competitors, who also run SAP on their cloud services.

“With COVID-19, we’re seeing more enterprises moving their IT infrastructure to the cloud, so they have better resiliency and scalability,” Rajput says. “For a lot of our customers, their biggest workload is enterprise resource planning (ERP) performed on SAP. If we can show them that moving to the cloud saves them money, then that will drive more cloud adoption.”

Options include making this an add-on to Microsoft Azure, says Amit Ganguli, a Microsoft Digital program management director based in Hyderabad. That also means possibly using Microsoft Azure Monitor, which now is in preview, or open-source code on GitHub.

For the team, making a big difference despite their few members has been a great source of satisfaction.

“I’m really proud of my team members,” Rajput says. “One of the strengths of Microsoft is it can quickly build teams that can solve big problems like this. I don’t feel like I’m just doing a job. What motivates me is that we’re having a positive impact on our customers.”

• Learn how Microsoft monitors SAP end to end.
• See how Microsoft monitors end-to-end enterprise health with Microsoft Azure.
• Check out how Microsoft migrated critical financial systems to Microsoft Azure.

The post How auto-scaling SAP on Microsoft Azure is benefitting Microsoft appeared first on Inside Track Blog.