7 citations · 11 across the 12 of their papers we have counts for
4 papers · 1 filter
Fast Weight Attention for Continual Learning
Yifan Zhang, Steve Ta, Jasper Zhang +8
Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule…
Unlocking Feature Learning in Gated Delta Networks at Scale
Yifeng Liu, Quanquan Gu
Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning metho…
Self-Distilled Policy Gradient
Yifeng Liu, Shiyuan Zhang, Yifan Zhang +1
On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward…
Deep Delta Learning
Yifan Zhang, Yifeng Liu, Mengdi Wang +1
Transformer residual streams are updated by addition. A sufficiently expressive residual block can represent content replacement, but the residual update itself has no operation th…