SAQL: A Stream-based Query System for Real-Time Abnormal System Behavior Detection

Abstract

Recently, advanced cyber attacks, which consist of a sequence of steps that involve many vulnerabilities and hosts, compromise the security of many well-protected businesses. This has led to the solutions that ubiquitously monitor system activities in each host (big data) as a series of events, and search for anomalies (abnormal behaviors) for triaging risky events. Since fighting against these attacks is a time-critical mission to prevent further damage, these solutions face challenges in incorporating expert knowledge to perform timely anomaly detection over the large-scale provenance data. To address these challenges, we propose a novel stream-based query system that takes as input, a real-time event feed aggregated from multiple hosts in an enterprise, and provides an anomaly query engine that queries the event feed to identify abnormal behaviors based on the specified anomalies. To facilitate the task of expressing anomalies based on expert knowledge, our system provides a domain-specific query language, SAQL, which allows analysts to express models for (1) rule-based anomalies, (2) time-series anomalies, (3) invariant-based anomalies, and (4) outlier-based anomalies. We deployed our system in NEC Labs America comprising 150 hosts and evaluated it using 1.1TB of real system monitoring data (containing 3.3 billion events). Our evaluations on a broad set of attack behaviors and micro-benchmarks show that our system has a low detection latency (<2s) and a high system throughput (110,000 events/s; supporting ~4000 hosts), and is more efficient in memory utilization than the existing stream-based complex event processing systems.

arXiv commentAccepted paper at USENIX Security Symposium 2018

References in corpus (1)

• AIQL: Enabling Efficient Attack Investigation from System Monitoring Data

Cited by in corpus (11)

• UNICORN: Runtime Provenance-Based Detector for Advanced Persistent Threats
• SIGL: Securing Software Installations Through Deep Graph Learning
• HyperService: Interoperability and Programmability Across Heterogeneous Blockchains
• Enabling Efficient Cyber Threat Hunting With Cyber Threat Intelligence
• Threat Detection and Investigation with System-level Provenance Graphs: A Survey
• Contextual Agent Security: A Policy for Every Purpose
• Runtime Analysis of Whole-System Provenance
• A System for Efficiently Hunting for Cyber Threats in Computer Systems Using Threat Intelligence
• Querying Streaming System Monitoring Data for Enterprise System Anomaly Detection
• Attentional Heterogeneous Graph Neural Network: Application to Program Reidentification
• A Query System for Efficiently Investigating Complex Attack Behaviors for Enterprise Security