Deploying Windows 10 at Microsoft as an in-place upgrade

As a core customer of the Windows Devices Group, Microsoft Digital deployed Windows 10 to 96,000 distributed users at Microsoft. The deployment included both remote users and users on the corporate network, and was completed in nine weeks. To improve on past operating-system deployments, Microsoft Digital deployed Windows 10 as an in-place upgrade that maintained productivity and offered a consistent user experience.

As a core customer of the Windows Devices Group, Microsoft Digital deployed Windows 10 to 96,000 distributed users at Microsoft more than three years ago. The deployment included both remote users and users on the corporate network and was completed in nine weeks. To improve on past operating-system deployments, Microsoft Digital deployed Windows 10 as an in-place upgrade that maintained productivity and offered a consistent user experience

Situation

Less than a year after the release of Windows 8.1 Update, Microsoft Digital deployed Windows 10. Increasingly rapid release cycles have made it necessary for Microsoft Digital to build a scalable approach to deploying this—and future—operating systems.

Microsoft Digital plays a primary role in using our own technology first at Microsoft. As such, Microsoft Digital partners with product teams early to test the software deployment experience.

Microsoft Digital gained useful insights into the company-wide deployment of operating systems through the progression from Windows 7, Windows 8, Windows 8.1, and now, Windows 10. Acquisitions also have given Microsoft Digital even more insight about the challenges that Microsoft customers face when deploying a new operating system into environments that use different applications and platforms.

Opportunities to improve the upgrade experience

A number of issues were addressed by the Windows 10 deployment.

• Complex environment. Most computers at Microsoft were running Windows 8.1 before the early adoption of Windows 10, although some computers were still running Windows 7 and Windows 8. A Microsoft acquisition came on board with most of its computers running Windows 7. The acquisition’s environment featured hundreds of line-of-business (LOB) applications that were developed to run on the Windows 7 client and its own System Center Configuration Manager infrastructure. They also relied on non-Microsoft solutions, including third-party endpoint protection, and a different VPN solution. Microsoft was in the process of assessing and migrating that environment when the Windows 10 deployment began.
• Increasingly rapid release cycles. The cadence of product releases is increasing. Microsoft Digital needed to build a scalable approach to migrate its global and distributed user base to Windows 10 in a phased manner.
• User readiness. When Microsoft Digital deployed Windows 7, users had multiple options for joining different domains and organizational units, setting up additional accounts, and provisioning their hard drives the way they wanted to. Those options were great for developers, but were complex and challenging for other users. Non-technical users would go to the installation point and not know what to do. It became crucial to make future upgrades easy for Microsoft Digital to deploy and for users to install.
• Empowering users and maintaining productivity. Rapid deployment processes also required creating a strong user experience that enables users to be productive, with rollback scenarios, self-service capabilities, and streamlined infrastructure that supports easy adoption.
• Remote users. About a third of full-time employees are mobile users all of the time, and roughly another third are mobile at any time that Microsoft Digital plans to do a deployment. Remote users may have either good internet connections or metered connections.
• Small offices without distribution points. During one acquisition, Microsoft Digital was faced with the challenge of improving a less than optimal corporate network installation experience. The existing System Center Configuration Manager infrastructure buildout only supported small packages and updates through Configuration Manager. Distribution Points were not placed in all office locations, so Microsoft Digital had to provide a good experience to about 5,000 users that would connect to the corporate network through a wide area network.

Learning from past deployments

With each operating-system deployment, issues arose that made Microsoft Digital think about how to streamline future deployments and increase adoption rates.

Windows 7

Microsoft Digital used a custom imaging solution during Windows 7 deployment, which was primarily developed to install the latest drivers and update BIOS on devices. Microsoft Digital used task sequence steps to update the BIOS and the drivers as part of the installation image used to deploy Windows 7 company-wide. It took nearly a year to get 80 percent of employees to install it.

Windows 8

When deploying Windows 8, Microsoft Digital made a conscious decision to improve the user experience. For Windows 7, remote users had to use installation media such as USB drives, CDs, and DVDs. To improve the installation experience for both remote and on-premises users, Microsoft Digital used the publicly available Microsoft Deployment Toolkit and created a Windows Installer front-end application that streamlined and simplified deployment. The application was designed to pick the best deployment solution for a user based on their location, including Microsoft Azure file service.

Adoption rates improved because it was easy for users to install. However, there was no significant decrease in help-desk call volume and it still took eight months to get 95 percent of employees upgraded to Windows 8.

Windows 8.1

The product group engaged Microsoft Digital to begin Windows 8.1 early adoption before the Windows 8 deployment was complete. To streamline deployment and promote adoption, Microsoft Digital collaborated with the product group to enable an in-place upgrade as a pilot for Windows 8.1. Using the in-place upgrade eliminated the need to build a complex zero touch deployment image. Microsoft Digital didn’t have to create packages to deal with data migration or application reinstallation—everything just worked. The upgrade had a 97 percent success rate, and in the few instances that the upgrade failed, it simply rolled the computer back to its previous operating system. Microsoft Digital saw a 35 percent reduction in help-desk calls for setup, which was over the 10 percent reduction goal. Microsoft Digital was able to deploy 95 percent of employees in three months by using the in-place upgrade and received user feedback that it was a great experience.

Solution

Windows 10 formally offers in-place upgrade as a deployment scenario. Users at Microsoft running Windows 7, Windows 8, and Windows 8.1 were upgraded to Windows 10 using the automated in-place upgrade method. In-place upgrade is the preferred deployment method for all but newly provisioned or rebuilt machines at Microsoft.

An early-adoption community helped validate user scenarios, and Microsoft Digital set its deployment target goal at 95 percent of users within 10 weeks of the Windows 10 release.

From an end-user perspective, installing Windows 10 with the in-place upgrade was very quick and easy. Users’ applications and data remained in place during the upgrade, and the small number of installation failures were simply rolled back to the previous operating system.

Microsoft Digital developed its deployment approach for Windows 10 to help:

• Better address compatibility
• Use the participation of early adoption communities
• Reduce the cost and management of deployments
• Improve user readiness and the user installation experience

The Windows 10 upgrade was made possible by using the Operating System Deployment (OSD) feature in Configuration Manager. It uses the same setup engine that has been trusted to install the Windows 10 update to hundreds of millions of consumer computers.

Maintaining data, applications, and settings and configurations

In-place upgrade maintains all user data, applications and settings from the previous operating system, so Microsoft Digital did not need to create or manage additional configuration scripts. For example, Microsoft Digital did not need to create or manage a script to maintain Power Management settings that had previously been implemented to help reduce power consumption. Those settings remained in effect after the operating system was upgraded to Windows 10.

Mandatory vs. optional installations

Configuration Manager supports mandatory and user-initiated installations

• User-initiated “pull” installations (optional). Microsoft Digital chose to use a self-service method, where users initiate the upgrade from the Software Center at their convenience.
• Mandatory “push” installations (required). Microsoft Digital can push the mandatory upgrade out to systems that are not upgraded by a certain date and time as a scheduled activity.

Users have the choice to “pull” and decide when to install the package. Microsoft Digital enabled an enforcement date to “push” the package out to users in a fairly seamless experience.

User-initiated upgrade

Microsoft Digital notified users that the Windows 10 upgrade was available to install by using the Configuration Manager application deployment model to send notifications to their desktop. The notification included the deadline for user-initiated deployments and a link that opened the Software Center.

When a user clicks the Install icon from the Software Center to initiate an upgrade installation of Windows 10:

• The computer connects to Configuration Manager and initiates the deployment.
• A policy request is sent to the management point.
• The Windows 10 installation package downloads. The download operates in the background, and visual notifications pop up to inform the user that the download has started.
• The Windows setup engine starts when the download is complete. It runs through a variety of compatibility checks to make sure there are no known issues with apps, drivers, BIOS versions—anything that could prevent installation completion. When it is done with the checks, the installer performs the update process.
• Windows setup updates Windows 7, Windows 8, or Windows 8.1 to Windows 10. The image is extracted onto the hard disk, moving the existing operating system out of the way as part of the process. The installer keeps all of the existing apps, user data, accounts, and settings, with the exception of some application files and settings that need to be moved aside and then merged back into the operating system.

Mandatory upgrades

Mandatory upgrades to Windows 10 were pushed out using Configuration Manager. One thing that Microsoft Digital found advantageous was scheduling the enforced upgrades on Tuesdays and Thursdays during the lunch hour (at noon). Microsoft Digital found that most users were comfortable starting a deployment as they were leaving for lunch. By the time users returned from lunch, they were running the next version of the operating system. Using a middle-of-the-week, middle-of-the-day strategy helped ensure that mandatory updates were pushed out during times when the most computers were connected to the corporate network.

Configuration Manager targeting

To help determine which machines to target for Windows 10 deployment, Microsoft Digital had to assess the environment. Configuration Manager Inventory was used to gather information about installed applications and driver versions. The primary purpose was to improve the user experience by identifying issues that had to be mitigated before deployment and/or omitting machines from the mandatory-upgrade package if there were known issues that would cause them to fail.

Task sequence wizard

With the Configuration Manager OSD task sequence wizard, Microsoft Digital configured pre-task, post-task, and on-failure task sequencing. It can be used to set a task for an action required after a failure. For example, if an upgraded component has failed back to Windows 7, but the user’s version of mobile VPN doesn’t work on Windows 7, a task can be set to reinstall the correct VPN client for that device.

To obtain the most up-to-date setup files, we used Dynamic Update during setup. As part of the task sequence we redirected machines to the public Windows Update (WU) servers for upgrade-specific content. This is because Windows setup-specific updates are available only via Windows Update. The updates to the operating system, the latest compatibility checks, and any needed driver updates were all retrieved from WU as part of the dynamic update process.

In some cases, using an application deployment to help the environment meet a specific condition can simplify deployment efforts and improve installation success. For example, there was an issue with an acquisition’s standard antivirus software. Microsoft Digital needed to replace its endpoint protection and migrate to Defender with the operating system. Microsoft Digital chose to use Configuration Manager Application Deployment as the most direct approach in this instance. By minimizing the number of sequence tasks used, Microsoft Digital was able to use the work the product team did to integrate OSD and Windows Setup. The Windows team invested a lot of time in Setup to ensure that it performed all the data migration tasks, and handled the applications migration, to help reduce the complexity of the Windows 10 deployment.

Pre-caching for remote users

There were some deployment failures that arose during the first set of 10,000 users participating in the Windows 10 early adoption program. Remote users were connected by DirectAccess, VPN, metered, or Wi-Fi connections. There was also a small population of users that don’t have a dedicated remote Distribution Point. About 50 percent of installation errors were due to users failing to download the package.

For mandatory deployments, pre-caching reduced installation errors and Microsoft Digital was able to improve the deployment success rate from 71 percent, for the first 10,000 early adopters, to 98 percent at product release by getting the package downloaded. The success-rate improvement was the combined result of enabling the pre-download packages and the setup team working on remediating issues.

To support five languages, Microsoft Digital had to create 10 packages for doing the deployment on x86 and x64 devices. Microsoft Digital chose not to pre-cache all 10 packages on the devices, and instead focused on pre-caching just the English US 64‑bit, because that represented the largest population of users.

Pre-caching solves a lot of problems for remote users. However, it doesn’t provide a solution for the small population of users that don’t have a dedicated remote Distribution Point. Early on, Microsoft Digital weighed its options for how to resolve that issue. Microsoft Digital chose to put the OS deployment package on an Azure file share and use application deployment to pre-cache the OS package on the device and still manage that deployment through Configuration Manager.

Application compatibility

From an application compatibility perspective, it was important for Microsoft Digital to not try to test everything. There were about 2,100 line-of-business applications at Microsoft. The program focused on testing only the business-critical applications identified by the business and used those results to determine if other similar applications (with shared code or similar design structure) needed to be tested.

By understanding shared dependencies, Microsoft Digital was able to target specific applications for testing that could represent several other applications in the application portfolio. If the application passed, the team could then hypothesize that similar applications would also pass. Using this approach, Microsoft Digital reduced the number of applications tested to only 20 percent of the LOB applications in its portfolio. Of these, around 8 percent of the applications were identified as primary and the remaining as secondary applications.

Over 96 percent of applications at Microsoft are browser-based, and Windows client-based applications account for almost 1 percent of the total apps. One of the primary compatibility issues Microsoft Digital saw was with browser-based apps that had incorrect compatibility view settings in the header and/or browser version detection.

In a test pass, Microsoft Digital achieved something that it had never experienced before: a 100 percent application compatibility test pass for Windows 10 with Internet Explorer 11. This success was partially due to using Internet Explorer 11 Enterprise Mode and Site List mode to get the docmode settings correct.

Microsoft Edge is the default browser in Windows 10. While the Microsoft Edge Product Group continues to enhance application compatibility work in future releases, websites that do not currently render or load in Microsoft Edge redirected to Internet Explorer 11 via the Enterprise Mode site list functionality. The websites that did not render were using ActiveX controls and Silverlight, which are not supported by Microsoft Edge.

Internet Explorer 11 compatibility

The Internet Explorer 11 Enterprise Mode (EmIE) feature was introduced for both Windows 7 and Windows 8.1.

EmIE can be implemented on devices using group policy to point to a location with the XML file. Sites can be added and it exercises Internet Explorer 8 code to run in the browser. This helped Microsoft Digital mitigate issues with LOB sites that were not running well in Internet Explorer 10 or 11.

This capability was extended in Windows 10 (on Windows 7, Windows 8.1 and Windows 10) by providing a way to override the docmode configured on any web page. For more information, see the blog “Announcing improvements to Enterprise Mode and Enterprise Site Discovery” link in the Resources section.

With EmIE, browser-based LOB sites could be set to come up in the right docmode, and functionality was retained without having to wait for app developers to fix their code. Microsoft Digital has found EmIE effective and easy to use, and it provides a great user experience.

Enterprise Site Discovery Toolkit

Enterprise Site Discovery Toolkit is a new feature in Internet Explorer that Microsoft Digital used to collect inventory information on the sites that users visit. The tool has been enhanced to allow targeting of specific domains or site zones, for example, only collecting data for the intranet sites.

The reporting includes information about what sites users are visiting, and Microsoft Digital compared those with the applications it was going to test. This also provided the docmode settings that are successful for the sites that can be used in EmIE.

During deployment, use of Enterprise Site Discovery was targeted to the North America region to comply with Microsoft legal guidance and global privacy regulations.

Virtual Machines for testing

Microsoft Digital no longer uses physical machines for application compatibility testing. A few years ago, Microsoft Digital set up a Virtual Machine (VM) farm for application engineers to run test passes on. Today, Microsoft Digital is providing over 300 VMs using Azurer VMs. This has allowed faster turnarounds for the results of a test pass and it allows retests on newer builds.

Centralized and distributed testing

Microsoft Digital currently uses both centralized and distributed testing. For centralized testing, a small group of testers uses a combination of automation, test tools, and best practices for rapid-release testing. They are able to go through the primary applications in about five days. Centralized testing frees up the application teams to focus on release cycles, and is used for targeted testing, where one application can represent several other applications in the portfolio.

Centralization has allowed Microsoft Digital to do incremental test passes on a regularly scheduled basis. And the centralized test team is shared across Microsoft Office early-adoption products and other early-adoption programs that require testing.

The distributed test teams are the experts on individual or groups of applications and can test them to a deeper level. For the Windows 10 deployment, the distributed test teams were given three weeks to test the secondary applications and report results. Distributed test teams have a better understanding of what the issues are and how to resolve them, but there are tradeoffs. Since those teams are also responsible for maintaining the applications, making changes and updates will interrupt maintenance and release schedules to do the testing to ensure that applications are compatible with the new technology.

Early adoption

Microsoft Digital created an early-adopter community that included the scores of Microsoft employees that enjoy being the first to install and use new technology. Early adopters provide valuable feedback and helps Microsoft Digital with scenario validation. An added benefit of the early adopter is that they often encourage other users to adopt the new products or technology. This has been a really great effort that required only a minimum of investment to build out a site and the architecture for users to participate.

Prior to product release, there were 38,000 users, roughly 40 percent of employees, internally running Windows 10. Microsoft Digital used the Windows Insider Program to make sure that early adopters were running the latest builds as they became available.

The early-adoption community is closely tied to a moderated internal community support forum, where users could report issues and seek assistance from other users. Microsoft Digital was able to gain early insight by watching the threads to identify issues as they surfaced.

User readiness and support

To encourage users to participate in the early-adoption program Microsoft Digital developed internal marketing materials that highlighted the ease with which users could install Windows 10 using the in-place upgrade. The materials highlighted the fact that user data and apps would not be affected by performing the upgrade.

Microsoft Digital created productivity guides to help users adjust to the new features in Windows 10 and worked with the help desk to train and prepare them for any support requests that came in.

Microsoft Digital used to only provide support through the help desk. Now, the moderated forum allows users to help other users (with a moderator) to ensure that their feedback is being heard.

Reporting

Microsoft Digital relies on reporting for enterprise operating system deployments. With a plan to track the key performance indicators, including adoption rates, Microsoft Digital was able to use different aspects of the information that was reported for different activities.

Pivot views of both organizational and geographical adoption rates were used to report the status of the Windows 10 deployment and to measure the success of regional or organizational unit initiatives, including target-based competitions, to encourage adoption.

SQL Server Reporting Services were used where organizational data is combined with inventory information collected by Configuration Manager and dynamic queries were used to create organizational-only level pivot views. IT managers used Microsoft Office Excel to access detailed reporting to help them resolve specific issues, but the information in those reports was not made generally available.

Reporting helps build target collection for mandatory deployments

To achieve better targeting of the machines for mandatory deployment of Windows 10, Microsoft Digital performed the following activities:

• Filtered the upgrade list to include only Microsoft employees’ computers.
• Identified business rules for determining which computers would be considered each user’s primary computer by using Configuration Manager data about installed operating systems and when the user was last logged in.
• For each deployment phase, queried Human Resources and Configuration Manager databases to identify users whose primary computers were already upgraded, and removing those users’ remaining computers from the upgrade list for that phase.

Troubleshooting

Error codes were reported to provide visibility and more data to help Microsoft Digital diagnose patterns and remediate certain issues. For example, if dozens of support calls come in for connection-related issues, Microsoft Digital can query to find information that the calls might have in common, such as whether calls are coming from a single building or geographic area. If the data shows that the problem is isolated to certain locations or conditions, they can act more quickly to remedy the situation.

Lessons learned

• Building an early adopter community was a good IT investment. Early adopters’ participation in installing, validating user scenarios, providing feedback, and sharing information in the moderated forums has been vital to the success of this and other product releases.
• The in-place upgrade method worked really well. All of the applications, data, and settings were processed across the migration. Microsoft users really didn’t have to do anything. Click, click, install, and they were up and running.
• The graceful return to the previous operating system in the event of a failure provided a better user experience. With Windows 10, Windows setup automatically creates a Windows.old folder containing the original operating system, applications, data, and settings. The Windows.old file copied the entire operating system, data, and settings. During a catastrophic failure, the Windows.old file allowed users to recover files and keys.
• Schedule mandatory updates to minimize user impact. Microsoft Digital ensured that mandatory updates were pushed out during days and times that the most computers would be connected to the corporate network. There was less impact on the help desk and it helped reduce support costs by roughly 50 percent.

Best practices

• Baseline Data. Use Configuration Manager to help gain an understanding of devices and applications in your environment
• Enable telemetry on devices in the environment. The Windows product group needs telemetry information to be able to maintain a good repository of data about enterprise applications. The group receives the most telemetry data about consumer applications, so its compatibility work is more focused in that space. By enabling telemetry data, enterprise apps are better represented in that database and more compatibility issues can be addressed before future products are released.
• Have daily sync meetings during the deployment to identify deployment issues and to determine what apps in the environment need to be remediated.

Compatibility and browser

• Focus testing on key business applications and use the results to include other applications with common technologies to broaden the testing pool, as needed.
• Use site discovery tools for browser-based application compatibility testing. It is much easier to remediate browser-based app issues using EmIE.
• Organizations should think about adding a centralized test team for LOB apps to perform faster and more efficient, broad testing to augment the deeper, more time-consuming tests that are performed by the distributed teams.
• Pre-caching improved the installation experience for remote users and users at locations without a dedicated distribution point.

Benefits

• Improved user experience. Users had a reliable operating-system upgrade that left their applications, data, and settings in place. They did not have to migrate data or reinstall applications after installing the Windows 10 upgrade. The rollback scenario ensured that users would remain productive with no data loss even if the upgrade failed.
• Speed of deployment and increased adoption rates. Building an installation image can take considerable effort, but an in-place upgrade package can be built in less than a day and yield better results. Windows 7 was deployed using a clean installation with migration, and it took nearly a year to deploy. Microsoft Digital deployed the Windows 10 upgrade to 85 percent of employees within four weeks of its release. The target of upgrading 95 percent of employees in 10 weeks was reached a week early. The Windows 10 deployment was the fastest company-wide operating system deployment Microsoft Digital has ever seen. In-place upgrade deployment using OSD was the key technology that enabled this successful deployment.
• Cost reduction. There was no image management required for the Windows 10 in-place upgrade, which helped reduce costs and streamlined the infrastructure.
• Reduction in help desk call volume. With Windows 10, Microsoft Digital saw a 50 percent reduction in help-desk call volume. As call volumes decreased, there was an increase in the moderated support forum use as employees used the forums to find the support and help that they needed. The moderated support forum was key in helping Microsoft Digital support users and also served as an early indicator, helping to quickly identify issues and remediate them.