Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
arXiv:2012.13091
Abstract
AlphaGo's astonishing performance has ignited an explosive interest in developing deep reinforcement learning (DRL) for numerous real-world applications, such as intelligent robotics. However, the often prohibitive complexity of DRL stands at the odds with the required real-time control and constrained resources in many DRL applications, limiting the great potential of DRL powered intelligent devices. While substantial efforts have been devoted to compressing other deep learning models, existing works barely touch the surface of compressing DRL. In this work, we first identify that there exists an optimal model size of DRL that can maximize both the test scores and efficiency, motivating the need for task-specific DRL agents. We therefore propose an Auto-Agent-Distiller (A2D) framework, which to our best knowledge is the first neural architecture search (NAS) applied to DRL to automatically search for the optimal DRL agents for various tasks that optimize both the test scores and efficiency. Specifically, we demonstrate that vanilla NAS can easily fail in searching for the optimal agents, due to its resulting high variance in DRL training stability, and then develop a novel distillation mechanism to distill the knowledge from both the teacher agent's actor and critic to stabilize the searching process and improve the searched agents' optimality. Extensive experiments and ablation studies consistently validate our findings and the advantages and general applicability of our A2D, outperforming manually designed DRL in both the test scores and efficiency. All the codes will be released upon acceptance.
References in corpus (32)
- Distilling the Knowledge in a Neural Network
- Continuous control with deep reinforcement learning
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- A Brief Survey of Deep Reinforcement Learning
- Neural Architecture Search with Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Categorical Reparameterization with Gumbel-Softmax
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- Prioritized Experience Replay
- Dueling Network Architectures for Deep Reinforcement Learning
- Efficient Neural Architecture Search via Parameter Sharing
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Noisy Networks for Exploration
- Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- Distributed Distributional Deterministic Policy Gradients
- SNAS: Stochastic Neural Architecture Search
- Model compression via distillation and quantization
- A Distributional Perspective on Reinforcement Learning
- Sample Efficient Actor-Critic with Experience Replay
- Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
- Transfer Learning in Deep Reinforcement Learning: A Survey
- Virtual to Real Reinforcement Learning for Autonomous Driving
- FasterSeg: Searching for Faster Real-time Semantic Segmentation
- AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks
- Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
- Kickstarting Deep Reinforcement Learning
- Distilling Policy Distillation
- Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations
- GOLD-NAS: Gradual, One-Level, Differentiable
- AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search