Approximating Language Model Training Data from Weights

John X. Morris, Junjie Oscar Yin, Woojeong Kim +2

Modern language models often have open weights but closed training data. We formalize the problem of data approximation from model weights and propose several baselines and metrics…