When Should We Introduce Safety Interventions During Pretraining?

Dylan Sam, Sachin Goyal, Pratyush Maini +2

Prior work has shown that safety interventions applied during pretraining, such as removing and rephrasing harmful content, can substantially improve the robustness of the resultin…