From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging

Qi Sun, Siyue Zhang, Yulin Chen +3

Training strong large language models (LLMs) requires high-quality supervision, which is often scarce. Recent work shows that paired preference data from weak-weaker model pairs (e…