PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learning

Abstract

The recent success of deep neural networks (DNNs) for function approximation in reinforcement learning has triggered the development of Deep Reinforcement Learning (DRL) algorithms in various fields, such as robotics, computer games, natural language processing, computer vision, sensing systems, and wireless networking. Unfortunately, DNNs suffer from high computational cost and memory consumption, which limits the use of DRL algorithms in systems with limited hardware resources. In recent years, pruning algorithms have demonstrated considerable success in reducing the redundancy of DNNs in classification tasks. However, existing algorithms suffer from a significant performance reduction in the DRL domain. In this paper, we develop the first effective solution to the performance reduction problem of pruning in the DRL domain, and establish a working algorithm, named Policy Pruning and Shrinking (PoPS), to train DRL models with strong performance while achieving a compact representation of the DNN. The framework is based on a novel iterative policy pruning and shrinking method that leverages the power of transfer learning when training the DRL model. We present an extensive experimental study that demonstrates the strong performance of PoPS using the popular Cartpole, Lunar Lander, Pong, and Pacman environments. Finally, we develop an open source software for the benefit of researchers and developers in related fields.

arXiv commentThis paper has been accepted for publication in the IEEE Journal of Selected Topics in Signal Processing

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• Playing Atari with Deep Reinforcement Learning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Do Deep Nets Really Need to be Deep?
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Predicting Parameters in Deep Learning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Deep Reinforcement Learning: An Overview
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks
• Deep Reinforcement Learning Radio Control and Signal Detection with KeRLym, a Gym RL Agent
• Deep Reinforcement One-Shot Learning for Artificially Intelligent Classification Systems

Cited by in corpus (5)

• Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
• Anomaly Search with Multiple Plays under Delay and Switching Costs
• GST: Group-Sparse Training for Accelerating Deep Reinforcement Learning
• Sparse Training for Federated Learning with Regularized Error Correction
• Ranking Policy Decisions