Server Engineering Insights for Large-Scale Online Services

The rapid growth of online services over the past decade has led to the development of large datacenters to host these workloads. The scale of online, user-facing services leads to unique design requirements in datacenter engineering and capacity provisioning. We explore these requirements specifically focusing on the design practices in system balancing, impact of technology trends, and the special challenges of online service workloads. Our study is based on an in-depth analysis of three very large-scale production Microsoft services: Hotmail, Cosmos, and Bing that together capture a wide range of characteristics of online services. We use this analysis to distill out key platform design practices and the motivations behind them. For instance, we investigate whether the Amdahl ratios computed over a decade ago for databases are relevant for online services. We observe significant deviations in the ratios due to the limitations of disk-based storage and the workarounds employed in these services. We also indicate how current technology trends such as multi-core chips and non-volatile memories are likely to steer these practices in the future.