Recording-Aware Evaluation of Multimodal OT Intrusion Detection

Abstract

Machine-learning-based intrusion detection systems (IDSs) for operational technology (OT) are commonly evaluated on benchmark datasets in which normal and attack data are collected in separate recordings. In such settings, high performance may reflect recording-associated differences as well as attack behavior. We introduce a recording-aware evaluation procedure that compares attack windows with background from the same recording, uses prior data for model fitting and calibration, tests temporal alignment against randomized event schedules, and reports event coverage together with alarm burden. On the ProvICS multimodal dataset, cyber modalities strongly distinguish the benign and attack recordings before the first labeled attack. Same-recording evaluation reverses the apparent modality ranking across all studied window sizes: the physical view becomes the strongest individual modality, while cyber-only fusion shows the largest difference between cross-recording and same-recording AUC, reaching 0.54. Operational analysis further shows that permissive event-overlap metrics can overstate detection when a small number of long alarm episodes cover multiple actions. An external analysis on HAI shows smaller and reference-dependent recording sensitivity. These findings show that cross-recording benchmark performance should be interpreted alongside same-recording, forward, temporal, and operational evaluations.

arXiv comment12 pages, 3 figures, 5 tables. Accepted for presentation at the Industrial Control System Security Workshop (ICSS 2026), held in conjunction with the IEEE Annual Computer Security Applications Conference (ACSAC 2026), Los Angeles, CA, USA, December 2026. Code and results are available at: https://github.com/ANTS-OntarioTechU/Recording-Aware-Evaluation-of-Multimodal-OT-Intrusion-Detection