An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay

Abstract

Prioritized Experience Replay (PER) is a deep reinforcement learning technique in which agents learn from transitions sampled with non-uniform probability proportionate to their temporal-difference error. We show that any loss function evaluated with non-uniformly sampled data can be transformed into another uniformly sampled loss function with the same expected gradient. Surprisingly, we find in some environments PER can be replaced entirely by this new loss function without impact to empirical performance. Furthermore, this relationship suggests a new branch of improvements to PER by correcting its uniformly sampled loss function equivalent. We demonstrate the effectiveness of our proposed modifications to PER and the equivalent loss function in several MuJoCo and Atari environments.

arXiv commentNeurIPS 2020

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Continuous control with deep reinforcement learning
• Soft Actor-Critic Algorithms and Applications
• Dueling Network Architectures for Deep Reinforcement Learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Distributed Prioritized Experience Replay
• Deep Reinforcement Learning that Matters
• A Deeper Look at Experience Replay
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Online Batch Selection for Faster Training of Neural Networks
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Variance Reduction in SGD by Distributed Importance Sampling
• Horizon: Facebook's Open Source Applied Reinforcement Learning Platform
• Off-Policy Policy Gradient with State Distribution Correction
• Diagnosing Bottlenecks in Deep Q-learning Algorithms
• Prioritized Sequence Experience Replay
• Planning by Prioritized Sweeping with Small Backups
• Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
• Reconciling $\lambda$-Returns with Experience Replay

Cited by in corpus (2)

• Convergent and Efficient Deep Q Network Algorithm
• Large Batch Experience Replay