Counterfactual Data Augmentation using Locally Factored Dynamics
arXiv:2007.02863
Abstract
Many dynamic processes, including common scenarios in robotic control and reinforcement learning (RL), involve a set of interacting subprocesses. Though the subprocesses are not independent, their interactions are often sparse, and the dynamics at any given time step can often be decomposed into locally independent causal mechanisms. Such local causal structures can be leveraged to improve the sample efficiency of sequence prediction and off-policy reinforcement learning. We formalize this by introducing local causal models (LCMs), which are induced from a global causal model by conditioning on a subset of the state space. We propose an approach to inferring these structures given an object-oriented state representation, as well as a novel algorithm for Counterfactual Data Augmentation (CoDA). CoDA uses local structures and an experience replay to generate counterfactual experiences that are causally valid in the global model. We find that CoDA significantly improves the performance of RL agents in locally factored tasks, including the batch-constrained and goal-conditioned settings.
arXiv commentIn Proceedings of NeurIPS 2020. 10 pages (+5 references, +12 appendix). Code available at \url{https://github.com/spitis/mrl}
References in corpus (40)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Gaussian Error Linear Units (GELUs)
- The Effectiveness of Data Augmentation in Image Classification using Deep Learning
- Addressing Function Approximation Error in Actor-Critic Methods
- Prioritized Experience Replay
- Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- Context-Specific Independence in Bayesian Networks
- Counterfactual Multi-Agent Policy Gradients
- MADE: Masked Autoencoder for Distribution Estimation
- Certifying Some Distributional Robustness with Principled Adversarial Training
- Visual Reinforcement Learning with Imagined Goals
- Reinforcement Learning with Augmented Data
- Benchmarking Model-Based Reinforcement Learning
- MONet: Unsupervised Scene Decomposition and Representation
- Quantifying Generalization in Reinforcement Learning
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- Relational Deep Reinforcement Learning
- Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Structured Disentangled Representations
- Weakly-Supervised Disentanglement Without Compromises
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
- SCAN: Learning Hierarchical Compositional Visual Concepts
- Recurrent Independent Mechanisms
- Contrastive Learning of Structured World Models
- COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
- Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning
- Deconfounding Reinforcement Learning in Observational Settings
- DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
- Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters
- Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
- ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay
- Object-Oriented Dynamics Predictor
- Causally Correct Partial Models for Reinforcement Learning
- Gradient-Based Neural DAG Learning
- Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning