Dhalion in action: automatic management of streaming applications

In a world where organizations are being inundated with data from various sources, analyzing data and gaining actionable insights in real-time has become a key service differentiator. Over the last few years, several stream processing frameworks have been developed to address the need for large-scale, real-time analytics. A crucial challenge in these environments is the complexity of configuring, managing and deploying long-running streaming applications. Operators must carefully tune these systems to balance competing objectives such as resource utilization and performance. At the same time, they must also account for external shocks such as unexpected load variations and service degradations.

In this demonstration, we show how operators can maintain healthy streaming applications without manual intervention while still meeting their performance objectives. We use Dhalion, an open-source library that sits on top of the streaming application, observes its behavior and automatically takes actions to keep the application in a healthy state. In particular, through various Dhalion policies that are configured by the attendees, we demonstrate how a streaming application can meet its performance objective by automatically configuring the amount of resources needed at the various application stages. We also demonstrate Dhalion’s modularity and extensibility that greatly simplifies the process of developing new policies which address different application requirements.