If It's Nice, Do It Twice: We Should Try Iterative Corpus Curation

Robin Young

Recent work demonstrates that filtering harmful content from pretraining data improves model safety without degrading capabilities. We propose a natural extension: do it again. A m…