UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

Xinlong Zhao, Dongsheng Liu, Hengyu Zhao +9

As available training data approaches its physical limit, gains from Scaling Laws have begun to diminish. Consequently, improving Large Language Models (LLMs) now depends less on d…