Decentralized Task-Aware Scheduling for Data Center Networks

Most data center applications perform rich and complex tasks (e.g., executing a search query or generating a user’s wall). From a network perspective, these tasks typically comprise multiple flows, which traverse different parts of the network at potentially different times. Existing network resource allocation schemes, however, treat all these flows in isolation – rather than as part of a task – and therefore only optimize flow-level metrics.

In this paper, we show that task-aware network scheduling, which groups flows of a task and schedules them together, can reduce both the average as well as tail completion time for typical data center applications. Based on the network footprint of real applications, we motivate the use of a scheduling policy that dynamically adapts the level of multiplexing in the network. To apply task-aware scheduling to online applications with small (sub-second) tasks, we design and implement Baraat, a decentralized task-aware scheduling system. Through experiments with Memcached on a small testbed and large-scale simulations, we show that Baraat can reduce tail task completion times by 60% for data analytics workloads and 40% for search workloads compared to existing mechanisms.