sBiLSAN: Stacked Bidirectional Self-attention LSTM Network for Anomaly Detection and Diagnosis from System Logs

High service availability is crucial for computer systems. Computer health diagnosis has become increasingly difficult due to a wide range of monitored information. Thus, it is essential to have the anomaly detection system along with firewalls and intrusion prevention systems. System logs are universally available in all the computer systems. The primary purpose of these logs is to record system states and events for debug and problem diagnosis. Therefore, efficient anomaly detection and prediction via log mining over unstructured texts are highly required. To this end, we seek to leverage machine learning based models to enhance the reliability of computer systems. This work aims to detect and predict system anomaly via stacked bidirectional self-attention long short-term memory networks in certain time intervals. In addition, we present a comprehensive study and evaluation on the existing anomaly detection algorithms, using a new large-scale benchmark consisting of both synthetic and real-world network traffic. Our evaluation and analysis indicate that our model can capture the complex representations of the anomaly, and obtain promising results as compared to the other state-of-the-art methods.