Log Message Anomaly Detection and Classification Using Auto-B/LSTM and Auto-GRU

Abstract

Log messages are now widely used in software systems. They are important for classification as millions of logs are generated each day. Most logs are unstructured which makes classification a challenge. In this paper, Deep Learning (DL) methods called Auto-LSTM, Auto-BLSTM and Auto-GRU are developed for anomaly detection and log classification. These models are used to convert unstructured log data to extracted features which is suitable for classification algorithms. They are evaluated using four data sets, namely BGL, Openstack, Thunderbird and IMDB. The first three are popular log data sets while the fourth is a movie review data set which is used for sentiment classification. The results obtained show that Auto-LSTM, Auto-BLSTM and Auto-GRU perform better than other well-known algorithms.

arXiv comment18 pages, 5 figures, 3 tables

References in corpus (6)

• Efficient Estimation of Word Representations in Vector Space
• Auto-Encoding Variational Bayes
• Recurrent Neural Network Regularization
• Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
• Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music