11 papers · 1 filter
Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling
Hyunji Lee, Wenhao Yu, Hongming Zhang +4
Hybrid models that combine state space models (SSMs) with attention mechanisms have shown strong performance by leveraging the efficiency of SSMs and the high recall ability of att…
Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact
Hyunji Lee, Seunghyun Yoon, Yunjae Won +7
Instruction tuning is a widely used approach to improve the instruction-following ability of large language models (LLMs). Instruction-tuning datasets typically include a mixture o…
Generative Prompt Internalization
Haebin Shin, Lei Ji, Yeyun Gong +3
Prompts used in recent large language model based applications are often fixed and lengthy, leading to significant computational overhead. To address this challenge, we propose Gen…
Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition
Jiyeon Kim, Hyunji Lee, Hyowon Cho +6
In this work, we investigate how a model's tendency to broadly integrate its parametric knowledge evolves throughout pretraining, and how this behavior affects overall performance,…
Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards
Hyeonbin Hwang, Doyoung Kim, Seungone Kim +2
Training on large amounts of rationales (i.e., CoT Fine-tuning) is effective at improving the reasoning capabilities of large language models (LLMs). However, acquiring human-autho…
Semiparametric Token-Sequence Co-Supervision
Hyunji Lee, Doyoung Kim, Jihoon Jun +4
In this work, we introduce a semiparametric token-sequence co-supervision training method. It trains a language model by simultaneously leveraging supervision from the traditional…