Interference and Generalization in Temporal Difference Learning

Abstract

We study the link between generalization and interference in temporal-difference (TD) learning. Interference is defined as the inner product of two different gradients, representing their alignment. This quantity emerges as being of interest from a variety of observations about neural networks, parameter sharing and the dynamics of learning. We find that TD easily leads to low-interference, under-generalizing parameters, while the effect seems reversed in supervised learning. We hypothesize that the cause can be traced back to the interplay between the dynamics of interference and bootstrapping. This is supported empirically by several observations: the negative relationship between the generalization gap and interference in TD, the negative effect of bootstrapping on interference and the local coherence of targets, and the contrast between the propagation rate of information in TD(0) versus TD($\lambda$) and regression tasks such as Monte-Carlo policy evaluation. We hope that these new findings can guide the future discovery of better bootstrapping methods.

arXiv commentSubmitted to ICML 2020. 20 pages, 14 figures

References in corpus (20)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Playing Atari with Deep Reinforcement Learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• On First-Order Meta-Learning Algorithms
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• On the importance of single directions for generalization
• A Study on Overfitting in Deep Reinforcement Learning
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Quantifying Generalization in Reinforcement Learning
• Essentially No Barriers in Neural Network Energy Landscape
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Assessing Generalization in Deep Reinforcement Learning
• Generalization and Regularization in DQN
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Towards Characterizing Divergence in Deep Q-Learning
• Stiffness: A New Perspective on Generalization in Neural Networks
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian
• Temporal Regularization in Markov Decision Process

Cited by in corpus (3)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• Decoupling Value and Policy for Generalization in Reinforcement Learning