5 papers
Learning Rate Transfer for Hybrid Transformer-SSM Architectures
Jimin Seo, Gyubok Lee, Yeonsik Jo +11
We study learning rate (LR) scaling for hybrid architectures combining Transformer and State-Space Model (SSM) blocks, a class adopted by several recent production language models.…
DCGC: Draft-Conditioned Global Correction for Complex Reasoning with Masked Diffusion Models
Minhae Oh, Nakyung Lee, Jungwoo Lee
Correcting flawed reasoning traces remains a significant challenge for Large Language Models (LLMs), whose autoregressive generation can propagate early mistakes into subsequent re…
Function-Level Execution Feedback for Code Preference Optimization
Idris Nechnech, Sehwan Kim, Jimin Seo +4
Process supervision has improved mathematical reasoning, where intermediate steps are naturally expressed as chains of thought. In code generation, however, process supervision rem…
Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation
Nakyung Lee, Yeongoon Kim, Minhae Oh +4
Transformer-based self-attention mechanism serves as the core of modern language models, yet it often suffers from localization, where attentions collapse onto a limited subset of…
RAISE: Enhancing Scientific Reasoning in LLMs via Step-by-Step Retrieval
Minhae Oh, Jeonghye Kim, Nakyung Lee +3
Scientific reasoning requires not only long-chain reasoning processes, but also knowledge of domain-specific terminologies and adaptation to updated findings. To deal with these ch…