10 citations · 10 across the 7 of their papers we have counts for
7 papers
RL Starts before RL: On Policy Distillation for Better Reinforcement Learning
Shuai Dong, Yongfu Zhu, Yuqi Xu +26
Reinforcement learning (RL) improves reasoning, but its performance depends on the policy from which training begins. We study on-policy distillation (OPD) as a preparation stage f…
What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation
Kanghui Tian, Siyuan Liu, Tianxiang Jiang +8
More privileged information does not always make a better teacher. We study this tension in on-policy self-distillation (OPSD), where a self-teacher scores the student's own rollou…
EG-MLA: Embedding-Gated Multi-head Latent Attention for Scalable and Efficient LLMs
Zhengge Cai, Haowen Hou
Reducing the key-value (KV) cache size is a crucial step toward enabling efficient inference in large language models (LLMs), especially under latency and memory constraints. While…
RWKV-X: A Linear Complexity Hybrid Language Model
Haowen Hou, Zhiyi Huang, Kaifeng Tan +2
In this paper, we introduce RWKV-X, a novel hybrid architecture that combines the efficiency of RWKV for short-range modeling with a sparse attention mechanism designed to capture…
RWKV-7 "Goose" with Expressive Dynamic State Evolution
Bo Peng, Ruichong Zhang, Daniel Goldstein +15
We present RWKV-7 "Goose", a new sequence modeling architecture with constant memory usage and constant inference time per token. Despite being trained on dramatically fewer tokens…
Delta-WKV: A Novel Meta-in-Context Learner for MRI Super-Resolution
Rongchang Lu, Bingcheng Liao, Haowen Hou +2
Magnetic Resonance Imaging (MRI) Super-Resolution (SR) addresses the challenges such as long scan times and expensive equipment by enhancing image resolution from low-quality input…
Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence
Bo Peng, Daniel Goldstein, Quentin Anthony +27
We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV (RWKV-4) architecture. Our architectural design advancements include multi-headed matrix-value…