14 papers
Multi-Head Attention Residuals
Cheng Luo, Zefan Cai, Junjie Hu
Transformers propagate information across depth through a single additive residual stream: every sublayer reads only the most recent state. Attention residuals relax this by lettin…
Test-Time Training with Next-Token Prediction
Xuan Ouyang, Zefan Cai, Junjie Hu
Next-token prediction is the self-supervised signal that trains language models, and every observed prompt token provides the same signal at test time. We study whether this signal…
MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Models
Haozhe Zhao, Zefan Cai, Shuzheng Si +5
Recent text-to-image models produce high-quality results but still struggle with precise visual control, balancing multimodal inputs, and requiring extensive training for complex m…
CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoning
Congmin Zheng, Jiachen Zhu, Jianghao Lin +6
Process Reward Models (PRMs) play a central role in evaluating and guiding multi-step reasoning in large language models (LLMs), especially for mathematical problem solving. Howeve…
GRASP: Graph Agentic Search over Propositions for Multi-hop Question Answering
Stockton Jenkins, Ramya Korlakai Vinayak, Junjie Hu
Agentic retrieval improves multi-hop question answering by giving language models autonomy to iteratively gather evidence. Recent work augments these systems with knowledge graphs…
Delta Attention Residuals
Cheng Luo, Zefan Cai, Junjie Hu
Attention Residuals replace standard additive residual connections with learned softmax attention over previous layer outputs, enabling selective cross-layer routing. However, stan…