Recomposing the Reinforcement Learning Building Blocks with Hypernetworks
arXiv:2106.06842
Abstract
The Reinforcement Learning (RL) building blocks, i.e. Q-functions and policy networks, usually take elements from the cartesian product of two domains as input. In particular, the input of the Q-function is both the state and the action, and in multi-task problems (Meta-RL) the policy can take a state and a context. Standard architectures tend to ignore these variables' underlying interpretations and simply concatenate their features into a single vector. In this work, we argue that this choice may lead to poor gradient estimation in actor-critic algorithms and high variance learning steps in Meta-RL algorithms. To consider the interaction between the input variables, we suggest using a Hypernetwork architecture where a primary network determines the weights of a conditional dynamic network. We show that this approach improves the gradient approximation and reduces the learning step variance, which both accelerates learning and improves the final performance. We demonstrate a consistent improvement across different locomotion tasks and different algorithms both in RL (TD3 and SAC) and in Meta-RL (MAML and PEARL).
arXiv commentICML 2021
References in corpus (22)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Spectral Normalization for Generative Adversarial Networks
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Training Very Deep Networks
- A Simple Neural Attentive Meta-Learner
- Model-Based Reinforcement Learning for Atari
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
- A Deeper Look at Experience Replay
- Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
- The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions
- Principled Weight Initialization for Hypernetworks
- A Closer Look at Deep Policy Gradients
- D2RL: Deep Dense Architectures in Reinforcement Learning
- HyperNetworks with statistical filtering for defending adversarial examples
- On approximating with neural networks
- Continual Model-Based Reinforcement Learning with Hypernetworks
- Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
- A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks