ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws

Ruihang Li, Yixuan Wei, Miaosen Zhang +3

High-quality data is crucial for the pre-training performance of large language models. Unfortunately, existing quality filtering methods rely on a known high-quality dataset as re…