75 citations · 241 across the 34 of their papers we have counts for
23 papers · 1 filter
Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models
Xing Chen, Hengshuai Yao
Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization impl…
GAIN: Multiplicative Modulation for Domain Adaptation
Hengshuai Yao, Xing Chen, Ahmed Murtadha +1
Adapting LLMs to new domains causes forgetting because standard methods (e.g., full fine-tuning, LoRA) inject new directions into the weight space. We show that forgetting is gover…
Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection
Hengshuai Yao, Xing Chen, Ahmed Murtadha +1
Standard Transformer attention uses identical dimensionality for queries, keys, and values, yet these components serve different roles: queries and keys produce scalar attention we…
Baird Counterexample is Solved: with an example of How to Debug a Two-time-scale Algorithm
Hengshuai Yao
Baird counterexample was proposed by Leemon Baird in 1995, first used to show that the Temporal Difference (TD(0)) algorithm diverges on this example. Since then, it is often used…
Careful at Estimation and Bold at Exploration
Xing Chen, Yijun Liu, Zhaogeng Liu +3
Exploration strategies in continuous action space are often heuristic due to the infinite actions, and these kinds of methods cannot derive a general conclusion. In prior work, it…
A new Gradient TD Algorithm with only One Step-size: Convergence Rate Analysis using - Smoothness
Hengshuai Yao
Gradient Temporal Difference (GTD) algorithms (Sutton et al., 2008, 2009) are the first ( is the number features) algorithms that have convergence guarantees for off-poli…
The Vanishing Decision Boundary Complexity and the Strong First Component
Hengshuai Yao
We show that unlike machine learning classifiers, there are no complex boundary structures in the decision boundaries for well-trained deep models. However, we found that the compl…
Class Interference of Deep Neural Networks
Dongcui Diao, Hengshuai Yao, Bei Jiang
Recognizing and telling similar objects apart is even hard for human beings. In this paper, we show that there is a phenomenon of class interference with all deep neural networks.…
Learning to Accelerate by the Methods of Step-size Planning
Hengshuai Yao
Gradient descent is slow to converge for ill-conditioned problems and non-convex problems. An important technique for acceleration is step-size adaptation. The first part of this p…
The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure
Xing Chen, Dongcui Diao, Hechang Chen +7
The popular Proximal Policy Optimization (PPO) algorithm approximates the solution in a clipped policy space. Does there exist better policies outside of this space? By using a nov…
Exploring the Training Robustness of Distributional Reinforcement Learning against Noisy State Observations
Ke Sun, Yingnan Zhao, Shangling Jui +1
In real scenarios, state observations that an agent observes may contain measurement errors or adversarial noises, misleading the agent to take suboptimal actions or even collapse…
Breaking the Deadly Triad with a Target Network
Shangtong Zhang, Hengshuai Yao, Shimon Whiteson
The deadly triad refers to the instability of a reinforcement learning algorithm when it employs off-policy learning, function approximation, and bootstrapping simultaneously. In t…
Variance-Reduced Off-Policy Memory-Efficient Policy Search
Daoming Lyu, Qi Qi, Mohammad Ghavamzadeh +3
Off-policy policy optimization is a challenging problem in reinforcement learning (RL). The algorithms designed for this problem often suffer from high variance in their estimators…
Towards a practical measure of interference for reinforcement learning
Vincent Liu, Adam White, Hengshuai Yao +1
Catastrophic interference is common in many network-based learning systems, and many proposals exist for mitigating it. But, before we overcome interference we must understand it b…
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
Shangtong Zhang, Bo Liu, Hengshuai Yao +1
We present the first provably convergent two-timescale off-policy actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is the introduction of a new critic,…
Is Fast Adaptation All You Need?
Khurram Javed, Hengshuai Yao, Martha White
Gradient-based meta-learning has proven to be highly effective at learning model initializations, representations, and update rules that allow fast adaptation from a few samples. T…
Hill Climbing on Value Estimates for Search-control in Dyna
Yangchen Pan, Hengshuai Yao, Amir-massoud Farahmand +1
Dyna is an architecture for model-based reinforcement learning (RL), where simulated experience from a model is used to update policies or value functions. A key component of Dyna…
Distributional Reinforcement Learning for Efficient Exploration
Borislav Mavrin, Shangtong Zhang, Hengshuai Yao +3
In distributional reinforcement learning (RL), the estimated distribution of value function models both the parametric and intrinsic uncertainties. We propose a novel and efficient…
Deep Reinforcement Learning with Decorrelation
Borislav Mavrin, Hengshuai Yao, Linglong Kong
Learning an effective representation for high-dimensional data is a challenging problem in reinforcement learning (RL). Deep reinforcement learning (DRL) such as Deep Q networks (D…
QUOTA: The Quantile Option Architecture for Reinforcement Learning
Shangtong Zhang, Borislav Mavrin, Linglong Kong +2
In this paper, we propose the Quantile Option Architecture (QUOTA) for exploration based on recent advances in distributional reinforcement learning (RL). In QUOTA, decision making…
ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search
Shangtong Zhang, Hao Chen, Hengshuai Yao
In this paper, we propose an actor ensemble algorithm, named ACE, for continuous control with a deterministic policy in reinforcement learning. In ACE, we use actor ensemble (i.e.,…
Negative Log Likelihood Ratio Loss for Deep Neural Network Classification
Donglai Zhu, Hengshuai Yao, Bei Jiang +1
In deep neural network, the cross-entropy loss function is commonly used for classification. Minimizing cross-entropy is equivalent to maximizing likelihood under assumptions of un…
Preconditioned Temporal Difference Learning
Yao HengShuai
This paper has been withdrawn by the author. This draft is withdrawn for its poor quality in english, unfortunately produced by the author when he was just starting his science rou…