4 papers
EG-MLA: Embedding-Gated Multi-head Latent Attention for Scalable and Efficient LLMs
Zhengge Cai, Haowen Hou
Reducing the key-value (KV) cache size is a crucial step toward enabling efficient inference in large language models (LLMs), especially under latency and memory constraints. While…
RWKV-X: A Linear Complexity Hybrid Language Model
Haowen Hou, Zhiyi Huang, Kaifeng Tan +2
In this paper, we introduce RWKV-X, a novel hybrid architecture that combines the efficiency of RWKV for short-range modeling with a sparse attention mechanism designed to capture…
RWKV-7 "Goose" with Expressive Dynamic State Evolution
Bo Peng, Ruichong Zhang, Daniel Goldstein +15
We present RWKV-7 "Goose", a new sequence modeling architecture with constant memory usage and constant inference time per token. Despite being trained on dramatically fewer tokens…
Delta-WKV: A Novel Meta-in-Context Learner for MRI Super-Resolution
Rongchang Lu, Bingcheng Liao, Haowen Hou +2
Magnetic Resonance Imaging (MRI) Super-Resolution (SR) addresses the challenges such as long scan times and expensive equipment by enhancing image resolution from low-quality input…