What bugs cause production cloud incidents?

Cloud services have become the backbone of today’s computing world. Runtime incidents, which adversely affect the expected service operations, are extremely costly in terms of user impacts and engineering efforts required to resolve them. Hence, such incidents are the target of much research effort. Unfortunately, there is limited understanding about cloud service incidents that actually happen during production runs: what cause them and how they are resolved. In this work, we carefully study hundreds of high severity incidents that occurred recently during the production runs of many Microsoft Azure services. We find software bugs to be a major cause behind these incidents, and make interesting observations about the types of software bugs that cause cloud incidents and how these bug-related incidents are resolved, providing motivation and guidance to future research in tackling cloud bugs and improving the cloud-service availability.