On Catastrophic Interference in Atari 2600 Games

Abstract

Model-free deep reinforcement learning is sample inefficient. One hypothesis -- speculated, but not confirmed -- is that catastrophic interference within an environment inhibits learning. We test this hypothesis through a large-scale empirical study in the Arcade Learning Environment (ALE) and, indeed, find supporting evidence. We show that interference causes performance to plateau; the network cannot train on segments beyond the plateau without degrading the policy used to reach there. By synthetically controlling for interference, we demonstrate performance boosts across architectures, learning algorithms and environments. A more refined analysis shows that learning one segment of a game often increases prediction errors elsewhere. Our study provides a clear empirical link between catastrophic interference and sample efficiency in reinforcement learning.

First two authors contributed equally. Code available to reproduce experiments at https://github.com/google-research/google-research/tree/master/memento

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Playing Atari with Deep Reinforcement Learning
• Prioritized Experience Replay
• Continual Learning Through Synaptic Intelligence
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Gradient Episodic Memory for Continual Learning
• Massively Parallel Methods for Deep Reinforcement Learning
• Exploration by Random Network Distillation
• Unifying Count-Based Exploration and Intrinsic Motivation
• A Distributional Perspective on Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Go-Explore: a New Approach for Hard-Exploration Problems
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Generalization and Regularization in DQN
• Neurogenesis Deep Learning
• Continual Unsupervised Representation Learning
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
• Divide-and-Conquer Reinforcement Learning

Cited by in corpus (2)

• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
• TOMA: Topological Map Abstraction for Reinforcement Learning