7 citations · 11 across the 12 of their papers we have counts for
12 papers
Fast Weight Attention for Continual Learning
Yifan Zhang, Steve Ta, Jasper Zhang +8
Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule…
Unlocking Feature Learning in Gated Delta Networks at Scale
Yifeng Liu, Quanquan Gu
Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning metho…
Self-Distilled Policy Gradient
Yifeng Liu, Shiyuan Zhang, Yifan Zhang +1
On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward…
Deep Delta Learning
Yifan Zhang, Yifeng Liu, Mengdi Wang +1
Transformer residual streams are updated by addition. A sufficiently expressive residual block can represent content replacement, but the residual update itself has no operation th…
Group Representational Position Encoding
Yifan Zhang, Zixiang Chen, Yifeng Liu +6
We present GRAPE (Group Representational Position Encoding), a unified framework for positional encoding based on group actions. GRAPE unifies two families of mechanisms: (i) multi…
Robust Layerwise Scaling Rules by Proper Weight Decay Tuning
Zhiyuan Fan, Yifeng Liu, Qingyue Zhao +2
Empirical scaling laws prescribe how to allocate parameters, data, and compute, while maximal-update parameterization (P) enables learning-rate transfer across widths by equaliz…
MARS-M: When Variance Reduction Meets Matrices
Yifeng Liu, Angela Yuan, Quanquan Gu
Matrix-based preconditioned optimizers, such as Muon, have recently been shown to be more efficient than scalar-based optimizers for training large-scale neural networks, including…
A Novel Approach to Differential Privacy with Alpha Divergence
Yifeng Liu, Zehua Wang
As data-driven technologies advance swiftly, maintaining strong privacy measures becomes progressively difficult. Conventional -differential privacy, while prevalent, exhib…
On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning
Yifan Zhang, Yifeng Liu, Huizhuo Yuan +3
Policy gradient algorithms have been successfully applied to enhance the reasoning capabilities of large language models (LLMs). KL regularization is ubiquitous, yet the design sur…
A Survey of Zero-Knowledge Proof Based Verifiable Machine Learning
Zhizhi Peng, Chonghe Zhao, Taotao Wang +7
Machine learning is increasingly deployed through outsourced and cloud-based pipelines, which improve accessibility but also raise concerns about computational integrity, data priv…
Tensor Product Attention Is All You Need
Yifan Zhang, Yifeng Liu, Huizhuo Yuan +4
Scaling language models to handle longer input sequences typically necessitates large key-value (KV) caches, resulting in substantial memory overhead during inference. In this pape…
MARS: Unleashing the Power of Variance Reduction for Training Large Models
Huizhuo Yuan, Yifeng Liu, Shuang Wu +2
Training deep neural networks--and more recently, large models demands efficient and scalable optimizers. Adaptive gradient algorithms like Adam, AdamW, and their variants have bee…