Mining Invariants from Console Logs for System Problem Detection

Detecting execution anomalies is very important to the maintenance and monitoring of large-scale distributed systems. People often use console logs that are produced by distributed systems for troubleshooting and problem diagnosis. However, manually inspecting con-sole logs for the detection of anomalies is unfeasible due to the increasing scale and complexity of distributed systems. Therefore, there is a great demand for automatic anomaly detection techniques based on log analysis. In this paper, we propose an unstructured log analysis technique for anomaly detection. In the technique, we propose a novel algorithm to automatically discover program invariants in logs. At first, we covert the unstructured logs to structured logs through a log parsing step. Then, we group the log messages according to the relationship among log parameters. After that, we learn the program invariants from the log message groups. The mined invariants can reveal the inherent linear characteristics of program work flows. With these learned invariants, we can automatically detect anomalies in logs. Experiments on Hadoop show that the technique can effectively detect execution anomalies. Compared with the state of art approaches, our approach can not only detect numerous real problems with high accuracy but also provide intuitive insight to the problems.