Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics

Abstract

Logs have been widely adopted in software system development and maintenance because of the rich runtime information they record. In recent years, the increase of software size and complexity leads to the rapid growth of the volume of logs. To handle these large volumes of logs efficiently and effectively, a line of research focuses on developing intelligent and automated log analysis techniques. However, only a few of these techniques have reached successful deployments in industry due to the lack of public log datasets and open benchmarking upon them. To fill this significant gap and facilitate more research on AI-driven log analytics, we have collected and released loghub, a large collection of system log datasets. In particular, loghub provides 19 real-world log datasets collected from a wide range of software systems, including distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. In this paper, we summarize the statistics of these datasets, introduce some practical usage scenarios of the loghub datasets, and present our benchmarking results on loghub to benefit the researchers and practitioners in this field. Up to the time of this paper writing, the loghub datasets have been downloaded for roughly 90,000 times in total by hundreds of organizations from both industry and academia. The loghub datasets are available at https://github.com/logpai/loghub.

Accepted by ISSRE 2023, Loghub datasets available at https://github.com/logpai/loghub

References in corpus (1)

• Length Matters: Clustering System Log Messages using Length of Words

Cited by in corpus (10)

• Log-based Anomaly Detection with Deep Learning: How Far Are We?
• Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection
• A Critical Review of Common Log Data Sets Used for Evaluation of Sequence-based Anomaly Detection Techniques
• A Survey on Automated Log Analysis for Reliability Engineering
• Studying Duplicate Logging Statements and Their Relationships with Code Clones
• PRINS: Scalable Model Inference for Component-based System Logs
• Log-based Anomaly Detection Without Log Parsing
• A Comprehensive Survey of Logging in Software: From Logging Statements Automation to Log Mining and Analysis
• Log Summarisation for Defect Evolution Analysis
• LogDP: Combining Dependency and Proximity for Log-based Anomaly Detection