Our customers are leading innovations across industries, and they are our strongest resource when it comes to security needs. One of the most significant blockers for customers is the risk assumed by connecting business-critical devices and equipment to the internet. Datacenters are a notable example. When you look at the datacenter’s essential infrastructure, the most critical functions of maintaining the environment have been intentionally kept offline to protect and preserve them. While the servers and network of a datacenter function as this powerful hub of innovation that drives global computing, the mechanical, and electrical systems that they depend on are, out of necessity, air gapped.

Mike Czamara, a General Manager at Microsoft, leads a team dedicated to the critical environment and availability of Azure Datacenters worldwide. “We approach datacenters with a necessarily conservative methodology. There’s the shell and there’s the critical space,” he says. Mike describes the shell as the building, the walls, the roof, the electrical system, the mechanical systems; everything that functions around the critical spaces or in service of them. The core is the servers and all the networking. The shell’s multiple systems operate simultaneously, but not always symbiotically since they are not digitally connected. Connecting critical equipment is a substantial risk for a datacenter focused on reducing, if not eliminating downtime.

However, disruptions happen. Outages happen. Mike’s team was finding that there were sometimes problems across building automation systems or power monitoring systems running code written by a third party. These issues sometimes lead to breakdowns. But, because the code at the heart of the issue was controlled by a third party, as Mike puts it, “Part of our destiny, and that of our customers, was out of our control.” Having greater control over the datacenter environment promised better outcomes for customers. The need for more control over the datacenter environment was nested in a larger challenge: the datacenter ecosystem itself.

Taking the first step

“We’re at the very beginning. We’re just walking up to the starting line. IoT was the first step,” says Mike. Really, the first step was an email. Adolfo Ferreira, a Senior Principal Technical Program Manager on Mike’s team, learned about Azure Sphere from the public announcement in April 2018. Adolfo immediately emailed Galen Hunt, the Managing Director of Azure Sphere. “I wrote him, begging him to give me a development kit. I told him what I wanted to do with it, and he took a kit away from one of his developers to give to me.” As Mike puts it, “From that point, it was game on.”

“Azure Sphere really triggered this big opportunity for us,” says Adolfo. At the time he discovered Azure Sphere, Adolfo and his team were looking to develop secured data acquisition from the mechanical and electrical systems, which have always been “read-only” systems. Azure Sphere gave them a way to securely connect these systems. The end-to-end solution includes secured hardware, the custom-built Azure Sphere OS, the cloud-based Azure Sphere Security Service, and ongoing servicing by Microsoft security experts for more than ten years. “I understood what Azure Sphere was trying to do, I knew the security was the highest level in the industry. I knew nothing could come close to the level of security Azure Sphere could offer,” says Adolfo.

For every Azure datacenter, security is the greatest priority, and the security requirements are spectacularly stringent. “Our data centers are not just running Microsoft’s businesses, but other tens of thousands of other company’s businesses within them. The Azure Sphere guardian module has layers and layers of security. The guardian module had no problem meeting our bar,” says Mike.

With Azure Sphere, the team started connecting mechanical and electrical systems—air handling units, power distribution units—to collect telemetry from the devices. In parallel, they started collecting data from servers and network devices. By using guardian modules powered by Azure Sphere, the team was able to confidently connect their most critical equipment when before the risk had been too great.

The team is exploring multiple scenarios that Azure Sphere has made possible. Maintenance, for example, is probably the most substantial commitment required of a datacenter. The standard approach is to have a regular, planned maintenance schedule to prevent problems. Sometimes it’s necessary, but often it’s just scheduled and so it just happens even when there’s no apparent need. Mike estimates that by staying on top of this sort of “blind maintenance” routine, only about 15 percent of maintenance will be reactive, meaning in response to an immediate need.

Informed by telemetry from connected systems, maintenance can become incisive, truly predictive, and can reduce reactive maintenance to as little as five percent. This can make a dramatic difference for organizations that forecast a budget one to five years out. Says Mike, “We are not spending money in hopes of preventing an outage. Our spend can become more targeted.”

Unlocking insights

Mike envisions a future of diagnostics in the datacenter. He sees a cache of information in every piece of equipment, “When we unlock that, it’s data that can create a wealth of knowledge. When I can see that a specific component in a certain generator is acting funny, and I can see how it affects performance health, I can make a more informed choice of what to do.” But he is thinking bigger than just generators or even just one datacenter. The knowledge gained from a single issue or incident in one datacenter can inform and improve performance for all the other datacenters located around the world.

But Mike is still thinking bigger than that—bigger than Microsoft. Having access to diverse sets of data, from partners and, maybe one day, from other organizations running equipment securely connected with Azure Sphere, can drive more informed decisions, and improve safety.

Smarter and safer

Mike’s team has been pioneering new safety measures enabled by Azure Sphere. Anytime a person must go into a datacenter to work on a piece of equipment, it is a point of risk. “There’s a problem of human error when a person goes into the wrong panel. They might turn off the wrong panel, which disrupts our customers.” In addition to the risk of uptime, there is also a serious risk to personal safety. Datacenters use a ton of power. A single datacenter uses between thirty-two and forty megawatts of power, roughly equivalent to six thousand homes. Panels have power sensors that will trip a warning siren when necessary, but a person’s instinctive reaction is to immediately shut the panel to turn off the alarm, potentially leaving problems unresolved. The team had to think about the problem, safety risk, and human behavior.

The team paired a klaxon siren with an Andon light and using a board built with Azure Sphere connected to the power sensor and datacenter control system. This setup made it possible to send the step-by-step of a work order, called a digital method of procedure (DMOP), directly to the panel requiring work. When a DMOP is released, the Andon light for the specific panel will change color to identify it as the panel requiring work. As the person goes through the DMOP for the work order, step by step, the light will reflect their progress. If the person misses a step, the light will signal the mistake and the klaxon will sound. Says Mike, “It’s exactly like bowling with bumpers.”

The team went a step further and integrated their electrical power monitoring system and their incident monitoring system. If a person working in the datacenter opens the wrong panel, a security alert is automatically sent, and a ticket is cut to a manager. “We immediately know when something has gone off-script if someone has put themselves or the datacenter in jeopardy. We can stop all work and figure out what’s going on,” says Mike.

Azure Sphere made it possible to securely coordinate multiple systems to create a new safety process. The connected panels do more than just help ensure correct and safe execution of processes, they also capture data when things go wrong so that the team can learn from incidents and resolve problems. “We’re creating systems that will keep us within the lines of safety and security and that help us adjust and refine those lines,” says Mike.

Impressive too is that Adolfo’s team developed the first of these safer electrical panels in only two months. “The Azure Sphere SDK made it possible for us to move fast and develop a complete solution from scratch, that was fully integrated with Azure Cloud Services,” he says. “With Azure Sphere, we can quickly turn any idea into a proof of concept.”

Strategic advantages

Adolfo’s team is focused on developing systems to increase reliability, security, and safety, and to optimize the building and systems that make up the “shell” of the datacenters. The total Azure Sphere offering, particularly the ongoing servicing by Microsoft security experts for more than ten years, has amplified the team’s ability to deliver business value. The cloud-based Azure Sphere Security Service automatically delivers OS and security updates to every device, so Adolfo and his team never have to worry about patching. “That’s all taken care of by Azure Sphere,” he says. And when the team needs to push new firmware to devices, Adolfo says it’s incredibly straightforward to do that at scale. Plus, Azure Sphere attestation guarantees the right firmware version is running on all their devices. “The services and support that Azure Sphere just provides have taken away the burden on my team,” he says.

Handling all that work at scale, especially security, would have required building out a dedicated team. “Having a whole team just for upkeep doesn’t actually add business value. Instead, we can spend our time on how to implement technology to improve availability, to reduce costs, to increase visibility into operations—that’s really how we add value. It’s a huge advantage. We have the opportunity to set the new standard in the datacenter industry, using Azure Sphere,” says Adolfo.

The business case for creativity

Mike sees the true value of Azure Sphere in how it enables innovation on a much larger scale of influence: “This tiny little thing is enabling us to evolve—not iterate anymore—evolve our space, our industry. It’s going to make our datacenters much more predictable, more usable, so that our customers reap the benefits and rewards of everything we’re doing.”

Mike started out by giving one engineer, Adolfo, total freedom to innovate with that first Azure Sphere development kit. Now Adolfo leads a team of ten whose only job is to create, to invent, to explore. “Because we were seeing such gains with one, two, then three people driving innovation, I was able to make a legitimate business case to bring on more people,” says Mike.

One of the reasons why Mike can confidently turn his team loose, without rails (“you can’t really have rails if you want to innovate,” he says), is because Azure Sphere offers a secured platform. The team’s grounding principles are safety, security, uptime, and cost. It must be safe. It must be secure. It cannot impact the customer. And it has to be affordable. Says Mike, “Azure Sphere delivers it all. It gives us this great foundation to work through wild ideas and opportunities.”

Get started with Azure Sphere today to build and test innovative, secured solutions for your organization, even while you’re working remotely.

The post IoT security: how Microsoft protects Azure Datacenters appeared first on Microsoft Security Blog.

The reality of securing IoT over time

It’s difficult to imagine any aspect of everyday life that isn’t affected by the influence of connectivity. The number of businesses that are using IoT is growing at a fast pace. By 2021, approximately 94 percent of businesses will be using IoT. Connectivity empowers organizations to unlock the full potential of the Internet of Things (IoT)—but it also introduces new cybersecurity attack vectors that they didn’t need to think about before. The reality is, connectivity comes at a cost: attackers with a wide range of motivations and skills are on the hunt, eager to exploit vulnerabilities or weak links in IoT. What does it take to manage those risks?

The cybersecurity threat landscape is ever evolving so a solution’s protection must also evolve regularly in order to remain effective. Securing a device is neither a one-time action nor is it a problem that is solely technical in nature. Implementing robust security measures upfront is not enough—risks need to be mitigated not just once, but constantly and throughout the full lifespan of a device. Facing this threat landscape ultimately means acknowledging that organizations will have to confront the consequences of attacks and newfound vulnerabilities. The question is, how to manage those risks beyond the technical measures that are in place?

A holistic approach to minimizing risk

Securing IoT devices against cyberattacks requires a holistic approach that complements up-front technical measures with ongoing practices that allow organizations to evaluate risks and establish a set of actions and policies that minimize threats over time. Cybersecurity is a multi-dimensional issue that requires the provider of an IoT solution to take several variables into account—it is not just the technology, but also the people who create and manage a product and the processes and practices they put in place, that will determine how resilient it is.

With Azure Sphere, we provide our customers with a robust defense that utilizes the evidence and learnings documented in the Seven Properties of Highly Secured Devices. One of the properties, renewable security, ensures that a device can update to a more secure state even after it has been compromised. As the threat landscape evolves, renewable security also enables us to counter new attack vectors through updates. This is essential, but not sufficient on its own. Our technology investments are enhanced through similar investments in security assurance and risk management that permeate all levels of an organization. The following sections highlight three key elements of our holistic approach to IoT security: continuous evaluation of our security promise, leveraging the power of the security community, and combining cyber and organizational resilience. 

Continuous evaluation of our security promise

All cyberattacks fall somewhere on a spectrum of complexity. On one side of the spectrum are simple and opportunistic attacks. Examples are off-the-shelf malware or attempts to steal data such as credentials. These attacks are usually performed by attackers with limited resources. On the opposite side of the spectrum are threat actors that use highly sophisticated methods to target specific parts of the system. Attackers within this category usually have many resources and can pursue an attack over a longer period of time. Given the multitude of threats across this spectrum, it is important to keep in mind that they all have one thing in common: an attacker faces relatively low risk with potentially very large rewards.

Taking this into account, we believe that in order to protect our customers we need to practice being our own worst enemy. This means our goal is to discover any vulnerabilities before the bad guys do. One proven approach is to test our solution from the same perspective as an attacker. So-called “red teams” are designed to emulate the attacks of adversaries, whereas “purple teams” perform both attacking and defending to harden a product from within.

Our approach to red team exercises is to try to mimic the threat landscape that devices are actually facing. We do this multiple times a year and across the full Azure Sphere stack. This means that our customers benefit from the rigorous security testing of our platform and are able to focus on the security of their own applications. We work with the world’s most renowned security service providers to test our product with a real-world attacker mentality for an extended period of time and from multiple perspectives. In addition, we leverage the full power of Microsoft internal security expertise to conduct regular internal red and purple team exercises. The practice of constantly evaluating our defense and emulating the ever-evolving threat landscape is an important part of our security hygiene—allowing us to find vulnerabilities, update all devices, and mitigate incidents before they even happen.

Leveraging the power of the security community

Another approach to finding vulnerabilities before attackers do is to engage with the cybersecurity community through bounty programs. We encourage security researchers with an interest in Azure Sphere to search for any vulnerabilities and we reward them for it. While our approach to red team exercises ensures regular testing of how we secure Azure Sphere, we also believe in the advantages of the continual and diverse assessment by anyone who is interested, at any point in time.

Security researchers play a significant role in securing our billions of customers across Microsoft, and we encourage the responsible reporting of vulnerabilities based on our Coordinated Vulnerability Disclosure (CVD). We invite researchers from across the world to look for and report any vulnerability through our Microsoft Azure Bounty Program. Depending on the quality of submissions and the level of severity, we award successful reports with up to $40,000 USD. We believe that researchers should be rewarded competitively when they improve the security of our platform, and we maintain these important relationships for the benefit of our customers.

From a risk management perspective, both red and purple team exercises and bug bounties are helpful tools to minimize the risk of attacks. But what happens when an IoT solution provider is confronted with a newly discovered security vulnerability? Not every organization has a cybersecurity incident response plan in place, and 77 percent of businesses do not have a consistently deployed plan. Finding vulnerabilities is important, but it is equally important to prepare employees and equip the organization with processes and practices that allow for a quick and efficient resolution as soon as a vulnerability is found.

Combining cyber and organizational resilience

Securing IoT is not just about preventing attackers from getting in; it’s also about how to respond when they do. Once the technical barrier has been passed, it is the resilience of the organization that the device has to fall back on. Therefore, it is essential to have a plan in place that allows your team to quickly respond and restore security. There are countless possible considerations and moving parts that must all fit together seamlessly as part of a successful cybersecurity incident response. Every organization is different and there is no one-size-fits-all, but a good place to start is with industry best practices such as the National Institute of Standards and Technology (NIST) Computer Security Incident Handling Guide. Azure Sphere’s standard operating procedures are aligned with those guidelines, in addition to leveraging Microsoft battle-tested corporate infrastructure.

Microsoft Security Response Center (MSRC) has been at the front line of security response for more than twenty years. Over time we have learned what it means to successfully protect our customers from harm from vulnerabilities in our products, and we are able to rapidly drive back attacks against our cloud infrastructure. Security researchers and customers are provided with an easy way to report any vulnerabilities and MSRC best-in-class security experts are monitoring communications 24/7 to make sure we can fix an issue as soon as possible.

Your people are a critical asset—when they’re educated on how to respond when an incident occurs, their actions can make all the difference. In addition to MSRC capabilities that are available at any time, we require everyone involved in security incident response to undergo regular and extensive training. Trust is easy to build when things are going right. What really matters in the long term is how we build trust when things go wrong. Our security response practices have been defined with that in mind.

Our commitment to managing the risks you are facing

The world will be more connected than it has ever been, and we believe this requires a strong, holistic, and ongoing focus on cybersecurity. Defending against today’s and tomorrow’s IoT threat landscape is not a static game. It requires continual assessment of our promise to secure your IoT solutions, innovation that improves our defense over time, and working with you and the security community. As the threat landscape evolves, so will we. Azure Sphere’s mission is to empower every organization on the planet to connect and create secured and trustworthy IoT devices. When you choose Azure Sphere, you can rely on our team and Microsoft to manage your risk so that you can focus on the true business value of your IoT solutions and products.

If you are interested in learning more about how Azure Sphere can help you securely unlock your next IoT innovation:

• Visit the Azure Sphere website to learn more.
• Get started.
• Secure your IoT deployment during the security talent shortage.

The post Managing risk in today’s IoT landscape: not a one-and-done appeared first on Microsoft Security Blog.