14 papers
The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training
Hongtao Zhang, Wenjie Zhou, Chenxi Jia +2
Large language model pre-training typically exhibits a two-phase trajectory: a fast initial loss drop followed by a prolonged slow improvement. We identify an underlying spectral p…
Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
Wenjie Zhou, Bohan Wang, Hongtao Zhang +3
Model merging has emerged as a lightweight paradigm for enhancing Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. In this work, we analyze lat…
Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs
Wanli Yang, Hongyu Zang, Junwei Zhang +5
Reinforcement learning (RL) has achieved remarkable success in LLM reasoning, but whether it can also improve direct recall of parametric knowledge remains an open question. We stu…
PromptCD: Test-Time Behavior Enhancement via Polarity-Prompt Contrastive Decoding
Baolong Bi, Yuyao Ge, Shenghua Liu +9
Reliable AI systems require large language models (LLMs) to exhibit behaviors aligned with human preferences and values. However, most existing alignment approaches operate at trai…
Gated Differentiable Working Memory for Long-Context Language Modeling
Lingrui Mei, Shenghua Liu, Yiwei Wang +7
Long contexts challenge transformers: attention scores dilute across thousands of tokens, critical information is often lost in the middle, and models struggle to adapt to novel pa…
A Survey of Vibe Coding with Large Language Models
Yuyao Ge, Lingrui Mei, Zenghao Duan +12
The advancement of large language models (LLMs) has catalyzed a paradigm shift from code generation assistance to autonomous coding agents, enabling a novel development methodology…