Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training

Zachary Speck, Asa Shepard

A single training example's contribution to a finished model is normally estimated rather than measured, because measuring it takes two expensive full pre-training runs that differ…