TUNA: Tuning Unstable and Noisy Cloud Applications

Autotuning plays a pivotal role in optimizing the performance of systems, particularly in large-scale cloud deployments, and has been used to improve the performance of a number of systems including databases, key-value stores, and operating systems. We find that one of the main challenges in performing autotuning in the cloud arises from performance variability or noise in system measurements. We first investigate the extent to which noise slows down autotuning and find that as little as 5% noise can lead to a 2.5x slowdown in converging to the best-performing configuration We also measure the magnitude of noise in cloud computing settings and find that, while some components (CPU, disk) have almost no performance variability there are still sources of significant variability (caches, memory). Additionally, we find that variability leads to autotuning finding unstable configurations, where for some workloads as many as 63.3% of configurations selected as “best” during tuning can degrade by 30% or more when deployed. Using this as motivation, this paper proposes a novel approach to improve the efficiency of autotuning systems by (a) detecting and removing outlier configurations, and (b) using ML-based approaches to provide a more stable true signal of de-noised experiment results to the optimizer. The resulting system, TUNA (Tuning Unstable and Noisy Cloud Applications) enables faster convergence and robust configurations. We find that configurations learned using TUNA perform better and with lower standard deviations during deployment, as compared to traditional sampling methodologies. Tuning PostgreSQL running an enterprise production workload, we find that TUNA can lead to 1.88x lower running time on average with 2.58𝑥 lower standard deviation compared to traditional sampling methodologies.  TUNA will be incorporated into the MLOS project.