Parameter Space Noise for Exploration

Abstract

Deep reinforcement learning (RL) methods generally engage in exploratory behavior through noise injection in the action space. An alternative is to add noise directly to the agent's parameters, which can lead to more consistent exploration and a richer set of behaviors. Methods such as evolutionary strategies use parameter perturbations, but discard all temporal structure in the process and require significantly more samples. Combining parameter noise with traditional RL methods allows to combine the best of both worlds. We demonstrate that both off- and on-policy methods benefit from this approach through experimental comparison of DQN, DDPG, and TRPO on high-dimensional discrete action environments as well as continuous control tasks. Our results show that RL with parameter noise learns more efficiently than traditional RL with action space noise and evolutionary strategies individually.

Updated to camera-ready ICLR submission

References in corpus (10)

• Prioritized Experience Replay
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Noisy Networks for Exploration
• Layer Normalization
• Deep Exploration via Bootstrapped DQN
• Count-Based Exploration with Neural Density Models
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning

Cited by in corpus (108)

• Noisy Networks for Exploration
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Exploration in Deep Reinforcement Learning: A Survey
• Large-Scale Study of Curiosity-Driven Learning
• Exploration by Random Network Distillation
• Agent57: Outperforming the Atari Human Benchmark
• Go-Explore: a New Approach for Hard-Exploration Problems
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Simple random search provides a competitive approach to reinforcement learning
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
• Review: Deep Learning in Electron Microscopy
• Unsupervised Learning of Object Keypoints for Perception and Control
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• A survey on intrinsic motivation in reinforcement learning
• Efficient Exploration via State Marginal Matching
• Proximal Distilled Evolutionary Reinforcement Learning
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Action Robust Reinforcement Learning and Applications in Continuous Control
• Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
• Deep reinforcement learning for large-eddy simulation modeling in wall-bounded turbulence
• Counting to Explore and Generalize in Text-based Games
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learning
• Learning Permutations with Sinkhorn Policy Gradient
• Whatever Does Not Kill Deep Reinforcement Learning, Makes It Stronger
• Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challenge
• WD3: Taming the Estimation Bias in Deep Reinforcement Learning
• Learning to Design Circuits
• Learning to Explore with Meta-Policy Gradient
• Learning latent state representation for speeding up exploration
• Exploration versus exploitation in reinforcement learning: a stochastic control approach
• Evolutionary Reinforcement Learning via Cooperative Coevolutionary Negatively Correlated Search
• Curiosity in exploring chemical space: Intrinsic rewards for deep molecular reinforcement learning
• The problem with DDPG: understanding failures in deterministic environments with sparse rewards
• Distributionally Robust Reinforcement Learning
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Self-Supervised Exploration via Disagreement
• Parallel Exploration via Negatively Correlated Search
• Efficient exploration with Double Uncertain Value Networks
• Variance Networks: When Expectation Does Not Meet Your Expectations
• Randomized Value Functions via Multiplicative Normalizing Flows
• Bayesian Policy Gradients via Alpha Divergence Dropout Inference
• Accelerated Deep Reinforcement Learning Based Load Shedding for Emergency Voltage Control
• Revisiting Design Choices in Proximal Policy Optimization
• Entropy Regularization for Mean Field Games with Learning
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• Reinforced Epidemic Control: Saving Both Lives and Economy
• Learning Awareness Models
• On the Weaknesses of Reinforcement Learning for Neural Machine Translation
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• Smooth Exploration for Robotic Reinforcement Learning
• Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement
• The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning
• Long-Term Visitation Value for Deep Exploration in Sparse Reward Reinforcement Learning
• On the Complexity of Exploration in Goal-Driven Navigation
• Privileged Information Dropout in Reinforcement Learning
• Variational Adaptive-Newton Method for Explorative Learning
• VFunc: a Deep Generative Model for Functions
• Attention-Privileged Reinforcement Learning
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Structured Control Nets for Deep Reinforcement Learning
• Run, skeleton, run: skeletal model in a physics-based simulation
• Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Sampling
• Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
• Challenges of Context and Time in Reinforcement Learning: Introducing Space Fortress as a Benchmark
• Learning Guidance Rewards with Trajectory-space Smoothing
• Implicit Generative Modeling for Efficient Exploration
• Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise
• Learning Agents With Prioritization and Parameter Noise in Continuous State and Action Space
• CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
• Competitive Experience Replay
• Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards
• Amplifying the Imitation Effect for Reinforcement Learning of UCAV's Mission Execution
• Learning to Play by Imitating Humans
• Learning Diverse Policies with Soft Self-Generated Guidance
• Novel Policy Seeking with Constrained Optimization
• Sample Efficient Ensemble Learning with Catalyst.RL
• Switching Isotropic and Directional Exploration with Parameter Space Noise in Deep Reinforcement Learning
• Stochastic Weight Matrix-based Regularization Methods for Deep Neural Networks
• Memristor Hardware-Friendly Reinforcement Learning
• Fractal AI: A fragile theory of intelligence
• On the Robustness of Deep Reinforcement Learning in IRS-Aided Wireless Communications Systems
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy
• ExTra: Transfer-guided Exploration
• Bootstrapped Q-learning with Context Relevant Observation Pruning to Generalize in Text-based Games
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Genetic-Gated Networks for Deep Reinforcement
• Adaptive Symmetric Reward Noising for Reinforcement Learning
• Function Approximation for Reinforcement Learning Controller for Energy from Spread Waves
• Evolutionary Stochastic Policy Distillation
• Learning Efficient and Effective Exploration Policies with Counterfactual Meta Policy
• Reinforcement Learning with Partial Parametric Model Knowledge
• Thompson Sampling via Local Uncertainty
• Can Q-learning solve Multi Armed Bantids?
• Progressive extension of reinforcement learning action dimension for asymmetric assembly tasks
• FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control
• Combine PPO with NES to Improve Exploration
• Theoretically Principled Deep RL Acceleration via Nearest Neighbor Function Approximation
• Reinforcement Learning for Robust Missile Autopilot Design
• Solving Atari Games Using Fractals And Entropy
• Learning Deep Energy Shaping Policies for Stability-Guaranteed Manipulation