When Probing Accuracy Saturates, Fragility Resolves: A Complementary Metric for LLM Pre-Training Analysis

Abstract

Standard linear probing declares a property "encoded" when a classifier on hidden states achieves high accuracy. The protocol works well on a snapshot but breaks across pre-training, with probe accuracy saturating within the first few thousand steps, leaving most of training invisible to the instrument. We introduce fragility, a complementary per-layer metric defined as the activation-noise level at which probe accuracy collapses. Fragility is sensitive to both the margin of separability and the redundancy of representation, both of which keep evolving long after accuracy plateaus. Applied to open-checkpoint language models, fragility recovers structure that accuracy alone cannot see. Moralized representations, our interest, emerge in stages, with high-accuracy detection of morally loaded words first and compositional encoding later. Because probe accuracy on its own tracks how lexically separable a dataset is, we establish the compositional encoding directly, by showing it transfers across construction types that share no contrast tokens. Where probing accuracy returns a flat answer, fragility returns a structured one. Raw critical noise also shows a layer-depth gradient, but we identify and correct a confound that, to our knowledge, has gone undocumented in noise-injection probing. That is, deeper layers carry larger activations, so a fixed absolute noise makes them look more robust than they are. Rescaling per layer removes most of the gradient, while within-layer comparisons stay valid and still separate identical-accuracy corpora by fragility. We recommend RMS-normalized critical noise for cross-layer claims and raw critical noise for within-layer comparisons.

25 pages, 6 figures. Code and datasets at https://github.com/deepsteer/deepsteer. v2: Adds an activation-scale (RMS-normalization) control, the v1 post-saturation critical-noise decline is withdrawn as a scale artifact; the layer-depth gradient is largely activation scale, with the residual not claimed as a genuine gradient. Finding 2 and the abstract are scoped