An Empirical Study of Observability Limits in Advanced Software Supply Chain Attacks

Abstract

Advanced software supply chain (SSC) attacks are increasingly runtime-only and leave fragmented evidence across hosts, services, and build/dependency layers, making any single telemetry stream insufficient for chain reconstruction. Despite this, no existing dataset provides multi-source runtime monitoring data with end-to-end chain-level ground truth for SSC attacks, leaving the observability limits of such attacks poorly understood. We present SynthChain, a multi-source runtime dataset with chain-level ground truth derived from real-world malicious packages and exploit campaigns, and use it to empirically study observability limits in SSC attacks. SynthChain covers seven representative SSC exploit scenarios across PyPI, npm, and C++ supply chains, spanning Windows, Linux, and containerized environments, with annotations for 14 MITRE ATT&CK tactics, 161 techniques, and 2,919 manually verified Indicators of Compromise (IOC) annotations across 22 log files spanning 11 telemetry types. Our observability analysis shows that no single source is chain-complete: even the best single source recovers fewer than 40% of expected attack steps. Fusing just two complementary sources improves reconstruction by roughly \(1.6\times\), but gains depend on which sources are combined rather than how many. We identify three systematic failure modes---missing-phase gaps, attribution breaks, and ambiguity---and derive telemetry planning guidelines that do not require prior knowledge of specific attacks. A preliminary sensitivity analysis confirms that the multi-source advantage persists under reduced per-source sampling rates. The corpus (approximately 0.59 million events) is released with ground truth and artifacts to support reproducible evaluation of runtime SSC defenses.