Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
arXiv:2103.02886
Abstract
Recent advances in off-policy deep reinforcement learning (RL) have led to impressive success in complex tasks from visual observations. Experience replay improves sample-efficiency by reusing experiences from the past, and convolutional neural networks (CNNs) process high-dimensional inputs effectively. However, such techniques demand high memory and computational bandwidth. In this paper, we present Stored Embeddings for Efficient Reinforcement Learning (SEER), a simple modification of existing off-policy RL methods, to address these computational and memory requirements. To reduce the computational overhead of gradient updates in CNNs, we freeze the lower layers of CNN encoders early in training due to early convergence of their parameters. Additionally, we reduce memory requirements by storing the low-dimensional latent vectors for experience replay instead of high-dimensional images, enabling an adaptive increase in the replay buffer capacity, a useful technique in constrained-memory settings. In our experiments, we show that SEER does not degrade the performance of RL agents while significantly saving computation and memory across a diverse set of DeepMind Control environments and Atari games.
References in corpus (19)
- Continuous control with deep reinforcement learning
- How transferable are features in deep neural networks?
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- DeepMind Control Suite
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Learning Latent Dynamics for Planning from Pixels
- Reinforcement Learning with Unsupervised Auxiliary Tasks
- Agent57: Outperforming the Atari Human Benchmark
- Equivalence Between Policy Gradients and Soft Q-Learning
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
- When to use parametric models in reinforcement learning?
- What is the State of Neural Network Pruning?
- FreezeOut: Accelerate Training by Progressively Freezing Layers
- SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
- Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
- Training CNNs with Selective Allocation of Channels