6 papers
Learning Rate Transfer for Hybrid Transformer-SSM Architectures
Jimin Seo, Gyubok Lee, Yeonsik Jo +11
We study learning rate (LR) scaling for hybrid architectures combining Transformer and State-Space Model (SSM) blocks, a class adopted by several recent production language models.…
DCGC: Draft-Conditioned Global Correction for Complex Reasoning with Masked Diffusion Models
Minhae Oh, Nakyung Lee, Jungwoo Lee
Correcting flawed reasoning traces remains a significant challenge for Large Language Models (LLMs), whose autoregressive generation can propagate early mistakes into subsequent re…
Mechanistic Circuit Identification for Controllable Data Generation
Nakyung Lee, Sangwoo Hong, Jungwoo Lee
While recent advances in data synthesis aim to curate high-quality datasets, most generation pipelines still rely on heuristic prompt-based control. This black-box paradigm provide…
Efficient Process Reward Modeling via Contrastive Mutual Information
Nakyung Lee, Sangwoo Hong, Jungwoo Lee
Recent research has devoted considerable effort to verifying the intermediate reasoning steps of chain-of-thought (CoT) trajectories using process reward models (PRMs) and other ve…
Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation
Nakyung Lee, Yeongoon Kim, Minhae Oh +4
Transformer-based self-attention mechanism serves as the core of modern language models, yet it often suffers from localization, where attentions collapse onto a limited subset of…
RAISE: Enhancing Scientific Reasoning in LLMs via Step-by-Step Retrieval
Minhae Oh, Jeonghye Kim, Nakyung Lee +3
Scientific reasoning requires not only long-chain reasoning processes, but also knowledge of domain-specific terminologies and adaptation to updated findings. To deal with these ch…