CausalMix: Data Mixture as Causal Inference for Language Model Training

Zinan Tang, Yukun Zhang, Shaomian Zheng +6

In Large Language Model (LLM) training, data mixing plays a pivotal role in determining model performance. Recent methods optimize mixture weights via proxy models, but they rely o…