Online Shift Detection and Conformal Adaptation for Deployed Safety Classifiers

Abstract

We study three failure modes of deployed safety classifiers: distribution shift, targeted evasion, and invalid scores from reasoning-model monitors. A reduced 800-cell subset of a pre-registered factorial (4 classifiers x 5 shifts x 20 seeds x 2 windows) gives 86.8% control-validated detection (alarm at or after onset, quiet paired no-shift stream; raw TPR 91.5%), mean latency 39.7 steps at w=100 and a classifier-by-shift crossover (${\eta }^2=0.186$ complete-case; 0.02-0.03 under a censoring sensitivity analysis) that motivates per-classifier monitoring. Shifted corpora are mostly harmful prompts, so this detects a constructed score change, not accuracy loss. The density-ratio step of weighted conformal adaptation collapses for 3/4 classifiers in high-dimensional embeddings; PCA to 32 dimensions raises temporal-shift coverage (+33pp, +20.5pp). A score-disagreement canary (an un-targeted classifier compared with the primary) flags 37/49 (75.5%) of the GCG inputs DeBERTa scores as safe (Llama Guard pair; 93.9% best single canary), leaving a residual that evades both. For one fixed attack objective we describe a local stall condition (gap $=1/(2\lambda )$; observed/predicted 0.86-1.00 at $\lambda \ge 2$, n=20; blocked rate does not replicate). Where both models judge content harmful there is no disagreement to detect (2%, 1/50). The best-value frontier LLM canary flags 83.7% (Wilson lower bound 71%) of 49 adversarial prompts with 7/1000 benign false positives. KS false-alarm rates inflate 5.4x under heterogeneous traffic; the two CUSUM runs there conflict. On o3, GCG-appended harmful inputs need 2.5x the token budget of simple benign inputs (T50 = 117 vs 46; length and content not separated), and at 16 tokens o3, o4-mini and gemini-2.5-pro all return 0/100 valid scores.

arXiv comment48 pages, 5 figures. v5 corrects v4: scan-martingale (CS) results regenerated and the CS-over-KS advantage withdrawn, KS fixed and 800-cell factorial regenerated, token-budget ratio 3.3x to 2.5x, ramp table and Pareto FPRs corrected, primary encoder is DeBERTa-v3-base, taxonomy and confidence-gating claims narrowed. Adds rotating probes, PAIR, CUSUM/EWMA