Triangle: Empowering Incident Triage with Multi-LLM-Agents

As cloud service systems grow in scale and complexity, incidents that indicate unplanned interruptions and outages become unavoidable. Rapid and accurate triage of these incidents to the appropriate responsible teams is crucial to maintain service reliability and prevent significant financial losses. However, existing incident triage methods relying on manual operations and predefined rules often struggle with efficiency and accuracy due to the heterogeneity of incident data and the dynamic nature of domain knowledge across multiple teams. To solve these issues, we propose Triangle, an end-to-end incident triage system based on a Multi-LLM-Agent framework. Triangle leverages a semantic distillation mechanism to tackle the issue of semantic heterogeneity in incident data, enhancing the accuracy of incident triage. Additionally, we introduce multi-role agents and a negotiation mechanism to emulate human engineers’ workflows, effectively handling decentralized and dynamic domain knowledge from multiple teams. Furthermore, our system incorporates an automated troubleshooting information collection and mitigation mechanism, reducing the reliance on human labor and enabling fully automated end-to-end incident triage. Extensive experiments conducted on real-world cloud production environment demonstrate that Triangle significantly improves the accuracy of incident triage more than 20% and reduces the time to engage about 3 time units per incident compared to state-of-the-art methods. Triangle has been successfully deployed in a system with tens of millions of users at a leading global technology company.