Prediction-Guided Design for Software Systems

While software system development is commonly conducted with explicit rules, machine learning (ML) has been driving a revolution in modern system design. In this paper, we introduce a new prediction-guided paradigm, which leverages ML techniques to support decision-makings for the system itself. In the proposed design, the system would be automatically driven by various type of data, e.g., system workloads, user behaviors, and platform operations, etc. More importantly, it brings a mindset of “proactive” to developers. Some significant issues can be thus eliminated before becoming catastrophe. In order to illustrate the benefits of the proposed paradigm, we present a project showcase, intelligent buffer management, which is used to achieve an optimal trade-off between having sufficiently large buffers to avoid failures and minimizing excess capacity in Microsoft Azure. It is designed in the prediction-guided paradigm to dynamically and proactively adjust the reserved buffer based on customer workload patterns and platform operations. The project not only significantly improves CFR (capacity fulfillment reliability) of tenant growth, but also reduces millions of dollars in COGS (cost of goods sold) for Microsoft.