A Survey on Industrial Control System Testbeds and Datasets for Security Research

Abstract

The increasing digitization and interconnection of legacy Industrial Control Systems (ICSs) open new vulnerability surfaces, exposing such systems to malicious attackers. Furthermore, since ICSs are often employed in critical infrastructures (e.g., nuclear plants) and manufacturing companies (e.g., chemical industries), attacks can lead to devastating physical damages. In dealing with this security requirement, the research community focuses on developing new security mechanisms such as Intrusion Detection Systems (IDSs), facilitated by leveraging modern machine learning techniques. However, these algorithms require a testing platform and a considerable amount of data to be trained and tested accurately. To satisfy this prerequisite, Academia, Industry, and Government are increasingly proposing testbed (i.e., scaled-down versions of ICSs or simulations) to test the performances of the IDSs. Furthermore, to enable researchers to cross-validate security systems (e.g., security-by-design concepts or anomaly detectors), several datasets have been collected from testbeds and shared with the community. In this paper, we provide a deep and comprehensive overview of ICSs, presenting the architecture design, the employed devices, and the security protocols implemented. We then collect, compare, and describe testbeds and datasets in the literature, highlighting key challenges and design guidelines to keep in mind in the design phases. Furthermore, we enrich our work by reporting the best performing IDS algorithms tested on every dataset to create a baseline in state of the art for this field. Finally, driven by knowledge accumulated during this survey's development, we report advice and good practices on the development, the choice, and the utilization of testbeds, datasets, and IDSs.

References in corpus (6)

• SCADA System Testbed for Cybersecurity Research Using Machine Learning Approach
• Learning from Mutants: Using Code Mutation to Learn and Monitor Invariants of a Cyber-Physical System
• Evaluation of Machine Learning-based Anomaly Detection Algorithms on an Industrial Modbus/TCP Data Set
• Constrained Concealment Attacks against Reconstruction-based Anomaly Detectors in Industrial Control Systems
• DAICS: A Deep Learning Solution for Anomaly Detection in Industrial Control Systems
• LICSTER -- A Low-cost ICS Security Testbed for Education and Research

Cited by in corpus (8)

• A Survey on Cyber-Resilience Approaches for Cyber-Physical Systems
• Network Intrusion Datasets: A Survey, Limitations, and Recommendations
• A Systematic Literature Review of Cyber Security Monitoring in Maritime
• SoK: Evaluations in Industrial Intrusion Detection Research
• Cyber-Physical Systems Security: A Comprehensive Review of Anomaly Detection Techniques
• SimProcess: High Fidelity Simulation of Noisy ICS Physical Processes
• ISIA-AF: Orchestrating Reproducible Attacks and Multi-Source Data Collection for OT Systems
• An Experimental Study of Machine Learning-Based Intrusion Detection for OPC UA over Industrial Private 5G Networks