5 papers · 1 filter
Towards Understanding Pause Token Fine-Tuning Dynamics: A Mode Retention Perspective
Jaehyeon Kim, Suhwan Kim, Nakyung Lee +4
Pause-token methods improve LLM reasoning by inserting special tokens into sequences. Prior work explains these gains through computational expressivity. However, there is relative…
DCGC: Draft-Conditioned Global Correction for Complex Reasoning with Masked Diffusion Models
Minhae Oh, Nakyung Lee, Jungwoo Lee
Correcting flawed reasoning traces remains a significant challenge for Large Language Models (LLMs), whose autoregressive generation can propagate early mistakes into subsequent re…
Efficient Process Reward Modeling via Contrastive Mutual Information
Nakyung Lee, Sangwoo Hong, Jungwoo Lee
Recent research has devoted considerable effort to verifying the intermediate reasoning steps of chain-of-thought (CoT) trajectories using process reward models (PRMs) and other ve…
Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation
Nakyung Lee, Yeongoon Kim, Minhae Oh +4
Transformer-based self-attention mechanism serves as the core of modern language models, yet it often suffers from localization, where attentions collapse onto a limited subset of…
RAISE: Enhancing Scientific Reasoning in LLMs via Step-by-Step Retrieval
Minhae Oh, Jeonghye Kim, Nakyung Lee +3
Scientific reasoning requires not only long-chain reasoning processes, but also knowledge of domain-specific terminologies and adaptation to updated findings. To deal with these ch…