Structure in Deep Reinforcement Learning: A Survey and Open Problems
arXiv:2306.16021 · doi:10.1613/jair.1.15703
Abstract
Reinforcement Learning (RL), bolstered by the expressive capabilities of Deep Neural Networks (DNNs) for function approximation, has demonstrated considerable success in numerous applications. However, its practicality in addressing various real-world scenarios, characterized by diverse and unpredictable dynamics, noisy signals, and large state and action spaces, remains limited. This limitation stems from poor data efficiency, limited generalization capabilities, a lack of safety guarantees, and the absence of interpretability, among other factors. To overcome these challenges and improve performance across these crucial metrics, one promising avenue is to incorporate additional structural information about the problem into the RL learning process. Various sub-fields of RL have proposed methods for incorporating such inductive biases. We amalgamate these diverse methodologies under a unified framework, shedding light on the role of structure in the learning problem, and classify these methods into distinct patterns of incorporating structure. By leveraging this comprehensive framework, we provide valuable insights into the challenges of structured RL and lay the groundwork for a design pattern perspective on RL research. This novel perspective paves the way for future advancements and aids in developing more effective and efficient RL algorithms that can potentially handle real-world scenarios better.
Published at the Journal of Artificial Intelligence Research, Volume 79, Pages 1167-1236
References in corpus (167)
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
- OpenAI Gym
- Segment Anything
- Conservative Q-Learning for Offline Reinforcement Learning
- Decision Transformer: Reinforcement Learning via Sequence Modeling
- A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
- Data-Efficient Hierarchical Reinforcement Learning
- Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning
- A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
- Stochastic Neural Networks for Hierarchical Reinforcement Learning
- Successor Features for Transfer in Reinforcement Learning
- Variational Intrinsic Control
- Meta-Reinforcement Learning of Structured Exploration Strategies
- Towards Deep Symbolic Reinforcement Learning
- Dream to Control: Learning Behaviors by Latent Imagination
- DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
- Approximate Policy Iteration with a Policy Language Bias: Solving Relational Markov Decision Processes
- Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
- Mastering Diverse Domains through World Models
- Provably Efficient Maximum Entropy Exploration
- Programmatically Interpretable Reinforcement Learning
- Diversity is All You Need: Learning Skills without a Reward Function
- Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
- Learning Invariant Representations for Reinforcement Learning without Reconstruction
- Contrastive Learning of Structured World Models
- Latent Space Policies for Hierarchical Reinforcement Learning
- DeepMDP: Learning Continuous Latent Space Models for Representation Learning
- Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
- Discrete Sequential Prediction of Continuous Actions for Deep RL
- Sharing Knowledge in Multi-Task Deep Reinforcement Learning
- Discovering Reinforcement Learning Algorithms
- Taming the Noise in Reinforcement Learning via Soft Updates
- Multi-Task Reinforcement Learning with Soft Modularization
- Improving Generalization in Meta Reinforcement Learning using Learned Objectives
- Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning
- MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies
- Model-Based Bayesian Reinforcement Learning in Large Structured Domains
- Provably efficient RL with Rich Observations via Latent State Decoding
- MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning
- Reinforcement Learning with Prototypical Representations
- Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
- Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
- Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
- Invariant Causal Prediction for Block MDPs
- FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs
- Exploration in Structured Reinforcement Learning
- What can I do here? A Theory of Affordances in Reinforcement Learning
- Entity Abstraction in Visual Model-Based Reinforcement Learning
- Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies
- Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
- One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
- Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimism
- An Explicitly Relational Neural Network Architecture
- Safe Option-Critic: Learning Safety in the Option-Critic Architecture
- Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
- Causal Reinforcement Learning using Observational and Interventional Data
- Human-Timescale Adaptation in an Open-Ended Task Space
- Learning Action Representations for Reinforcement Learning
- Multi-Task Reinforcement Learning with Context-based Representations
- PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making
- Patterns for Learning with Side Information
- Improving Intrinsic Exploration with Language Abstractions
- Discovered Policy Optimisation
- Harnessing Structures for Value-Based Planning and Reinforcement Learning
- A Survey of Knowledge-based Sequential Decision Making under Uncertainty
- A Survey on Interpretable Reinforcement Learning
- Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits
- Plannable Approximations to MDP Homomorphisms: Equivariance under Actions
- Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learning
- Deep Surrogate Assisted Generation of Environments
- Skill-based Meta-Reinforcement Learning
- Counterfactual Data Augmentation using Locally Factored Dynamics
- URLB: Unsupervised Reinforcement Learning Benchmark
- Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting
- Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning
- MICo: Improved representations via sampling-based state similarity for Markov decision processes
- Generalization to New Actions in Reinforcement Learning
- Faster and Safer Training by Embedding High-Level Knowledge into Deep Reinforcement Learning
- Policy Consolidation for Continual Reinforcement Learning
- Is Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for Hemodynamic Management in Sepsis Patients
- Approximate Linear Programming for First-order MDPs
- Reinforcement Learning in Rich-Observation MDPs using Spectral Methods
- Contextualize Me -- The Case for Context in Reinforcement Learning
- Active Exploration for Learning Symbolic Representations
- Causal Influence Detection for Improving Efficiency in Reinforcement Learning
- Hyperparameters in Reinforcement Learning and How To Tune Them
- Tell me why! Explanations support learning relational and causal structure
- Incorporating Relational Background Knowledge into Reinforcement Learning via Differentiable Inductive Logic Programming
- Symbolic Network: Generalized Neural Policies for Relational MDPs
- Geometric Entropic Exploration
- Solving Compositional Reinforcement Learning Problems via Task Reduction
- Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
- Modular Lifelong Reinforcement Learning via Neural Composition
- Exploration via Elliptical Episodic Bonuses
- Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machines
- Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewards
- Learning Markov State Abstractions for Deep Reinforcement Learning
- Motif: Intrinsic Motivation from Artificial Intelligence Feedback
- Denoised MDPs: Learning World Models Better Than the World Itself
- Robot Representation and Reasoning with Knowledge from Reinforcement Learning
- Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents
- Structured Control Nets for Deep Reinforcement Learning
- Interpretable Model-based Hierarchical Reinforcement Learning using Inductive Logic Programming
- Reachability Constrained Reinforcement Learning
- Interpretable Dynamics Models for Data-Efficient Reinforcement Learning
- Meta-Learning of Structured Task Distributions in Humans and Machines
- Towards an Interpretable Hierarchical Agent Framework using Semantic Goals
- Conditional Mutual Information for Disentangled Representations in Reinforcement Learning
- Learning Robust State Abstractions for Hidden-Parameter Block MDPs
- Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning
- Environment Generation for Zero-Shot Compositional Reinforcement Learning
- A Generalized Framework for Population Based Training
- Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions
- Automated Dynamic Algorithm Configuration
- Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions
- A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning
- Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare
- Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learning
- Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
- Relational Abstractions for Generalized Reinforcement Learning on Symbolic Problems
- Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning
- Model Primitive Hierarchical Lifelong Reinforcement Learning
- SMART: Self-supervised Multi-task pretrAining with contRol Transformers
- Building a Subspace of Policies for Scalable Continual Learning
- Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
- Factored World Models for Zero-Shot Generalization in Robotic Manipulation
- Bilinear value networks
- Explicitly Encouraging Low Fractional Dimensional Trajectories Via Reinforcement Learning
- Unsupervised Reinforcement Learning in Multiple Environments
- Reinforcement Learning with Subspaces using Free Energy Paradigm
- Bayesian Generational Population-Based Training
- Continual Learning of Control Primitives: Skill Discovery via Reset-Games
- Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
- The Benefits of Model-Based Generalization in Reinforcement Learning
- Block Contextual MDPs for Continual Learning
- Reinforcement Learning with Exogenous States and Rewards
- Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning
- Learning to Optimize for Reinforcement Learning
- Recursive Constraints to Prevent Instability in Constrained Reinforcement Learning
- Exploration by Maximizing Rényi Entropy for Reward-Free RL Framework
- Learning Two-Step Hybrid Policy for Graph-Based Interpretable Reinforcement Learning
- Graph Neural Networks for Relational Inductive Bias in Vision-based Deep Reinforcement Learning of Robot Control
- Reinforcement Learning in Linear MDPs: Constant Regret and Representation Selection
- Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPs
- Hierarchical Representation Learning for Markov Decision Processes
- A Kernel Perspective on Behavioural Metrics for Markov Decision Processes
- Compositional Multi-Object Reinforcement Learning with Linear Relation Networks
- TempLe: Learning Template of Transitions for Sample Efficient Multi-task RL
- Deep Laplacian-based Options for Temporally-Extended Exploration
- Sample Efficient Deep Reinforcement Learning via Local Planning
- Mesh-based Tools to Analyze Deep Reinforcement Learning Policies for Underactuated Biped Locomotion
- CompoSuite: A Compositional Reinforcement Learning Benchmark
- Disentangled (Un)Controllable Features
- Nuclear Norm Maximization Based Curiosity-Driven Learning
- Cell-Free Latent Go-Explore
- Reinforcement Learning in Factored Action Spaces using Tensor Decompositions
- Towards Deployable RL -- What's Broken with RL Research and a Potential Fix
- Meta-Reinforcement Learning via Exploratory Task Clustering
- Binding Actions to Objects in World Models
- AutoRL Hyperparameter Landscapes
- Feudal Graph Reinforcement Learning
- Heterogeneous Knowledge for Augmented Modular Reinforcement Learning
- Latent State Marginalization as a Low-cost Approach for Improving Exploration
- Safe Reinforcement Learning with Mixture Density Network: A Case Study in Autonomous Highway Driving