Log-based Anomaly Detection with Deep Learning: How Far Are We?

Abstract

Software-intensive systems produce logs for troubleshooting purposes. Recently, many deep learning models have been proposed to automatically detect system anomalies based on log data. These models typically claim very high detection accuracy. For example, most models report an F-measure greater than 0.9 on the commonly-used HDFS dataset. To achieve a profound understanding of how far we are from solving the problem of log-based anomaly detection, in this paper, we conduct an in-depth analysis of five state-of-the-art deep learning-based models for detecting system anomalies on four public log datasets. Our experiments focus on several aspects of model evaluation, including training data selection, data grouping, class distribution, data noise, and early detection ability. Our results point out that all these aspects have significant impact on the evaluation, and that all the studied models do not always work well. The problem of log-based anomaly detection has not been solved yet. Based on our findings, we also suggest possible future work.

arXiv commentAccepted by The 44th International Conference on Software Engineering (ICSE 2022)

References in corpus (4)

• Length Matters: Clustering System Log Messages using Length of Words
• Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection
• Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics
• A Survey on Automated Log Analysis for Reliability Engineering

Cited by in corpus (16)

• Deep Learning for Anomaly Detection in Log Data: A Survey
• RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models
• A Critical Review of Common Log Data Sets Used for Evaluation of Sequence-based Anomaly Detection Techniques
• On the Effectiveness of Log Representation for Log-based Anomaly Detection
• Multivariate Log-based Anomaly Detection for Distributed Database
• Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction
• A Comprehensive Study of Machine Learning Techniques for Log-Based Anomaly Detection
• Reducing Events to Augment Log-based Anomaly Detection Models: An Empirical Study
• A Comparative Study on Large Language Models for Log Parsing
• ADALog: Adaptive Unsupervised Anomaly detection in Logs with Self-attention Masked Language Model
• LLM meets ML: Data-efficient Anomaly Detection on Unstable Logs
• "I see models being a whole other thing": An Empirical Study of Pre-Trained Model Naming Conventions and A Tool for Enhancing Naming Consistency
• $K^4$: Online Log Anomaly Detection Via Unsupervised Typicality Learning
• A unified framework for detecting point and collective anomalies in operating system logs via collaborative transformers
• AUCAD: Automated Construction of Alignment Dataset from Log-Related Issues for Enhancing LLM-based Log Generation
• Using Large Language Models to Support Automation of Failure Management in CI/CD Pipelines: A Case Study in SAP HANA