Reinforcement Learning with Deep Energy-Based Policies

Abstract

We propose a method for learning expressive energy-based policies for continuous states and actions, which has been feasible only in tabular domains before. We apply our method to learning maximum entropy policies, resulting into a new algorithm, called soft Q-learning, that expresses the optimal policy via a Boltzmann distribution. We use the recently proposed amortized Stein variational gradient descent to learn a stochastic sampling network that approximates samples from this distribution. The benefits of the proposed algorithm include improved exploration and compositionality that allows transferring skills between tasks, which we confirm in simulated experiments with swimming and walking robots. We also draw a connection to actor-critic methods, which can be viewed performing approximate inference on the corresponding energy-based model.

References in corpus (4)

• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Learning and Transfer of Modulated Locomotor Controllers
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning

Cited by in corpus (181)

• Soft Actor-Critic Algorithms and Applications
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Learning Representations by Maximizing Mutual Information Across Views
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Conservative Q-Learning for Offline Reinforcement Learning
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Learning Optimal Resource Allocations in Wireless Systems
• Learning to Perform Local Rewriting for Combinatorial Optimization
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Review: Deep Learning in Electron Microscopy
• Generative Adversarial Imitation from Observation
• How to Train Your Energy-Based Models
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Active Learning in Robotics: A Review of Control Principles
• Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraints
• Multiagent Soft Q-Learning
• A reinforcement learning approach to rare trajectory sampling
• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• The Two Kinds of Free Energy and the Bayesian Revolution
• Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
• Learning from Few Samples: A Survey
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• A Reinforcement Learning Approach for Transient Control of Liquid Rocket Engines
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation
• Improved Contrastive Divergence Training of Energy Based Models
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
• Deep Reinforcement Learning amidst Lifelong Non-Stationarity
• Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning
• Predictive Information Accelerates Learning in RL
• Action and Perception as Divergence Minimization
• Exploration versus exploitation in reinforcement learning: a stochastic control approach
• Boosting Trust Region Policy Optimization by Normalizing Flows Policy
• Distributionally Robust Reinforcement Learning
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• Quantifying the effects of environment and population diversity in multi-agent reinforcement learning
• Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning
• A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Efficient (Soft) Q-Learning for Text Generation with Limited Good Data
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Behavior Priors for Efficient Reinforcement Learning
• Recurrent Off-policy Baselines for Memory-based Continuous Control
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• Making Sense of Reinforcement Learning and Probabilistic Inference
• Energy-Based Imitation Learning
• Connecting the Dots Between MLE and RL for Sequence Prediction
• Batch Policy Learning under Constraints
• A Tutorial on Sparse Gaussian Processes and Variational Inference
• Deep Reinforcement Learning for Clinical Decision Support: A Brief Survey
• Leveraging exploration in off-policy algorithms via normalizing flows
• Smoothed Action Value Functions for Learning Gaussian Policies
• Emergence of Theory of Mind Collaboration in Multiagent Systems
• State-only Imitation with Transition Dynamics Mismatch
• Boltzmann machines and energy-based models
• Disentangled Skill Embeddings for Reinforcement Learning
• Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
• Noise-Robust End-to-End Quantum Control using Deep Autoregressive Policy Networks
• Strictly Batch Imitation Learning by Energy-based Distribution Matching
• Learning-Driven Exploration for Reinforcement Learning
• Effective Exploration for Deep Reinforcement Learning via Bootstrapped Q-Ensembles under Tsallis Entropy Regularization
• On Entropy Regularized Path Integral Control for Trajectory Optimization
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
• Reinforcement Learning Your Way: Agent Characterization through Policy Regularization
• Generalized Second Order Value Iteration in Markov Decision Processes
• Can Interpretable Reinforcement Learning Manage Prosperity Your Way?
• EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based Models
• Stable Policy Optimization via Off-Policy Divergence Regularization
• Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm
• Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning
• Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
• Accelerating Reinforcement Learning Agent with EEG-based Implicit Human Feedback
• Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback
• Reinforcement Learning by Guided Safe Exploration
• Population-Guided Parallel Policy Search for Reinforcement Learning
• Learning to Robustly Negotiate Bi-Directional Lane Usage in High-Conflict Driving Scenarios
• Information Theoretic Model Predictive Q-Learning
• Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewards
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Two-stage Deep Reinforcement Learning for Inverter-based Volt-VAR Control in Active Distribution Networks
• A Survey of Exploration Methods in Reinforcement Learning
• Trajectory Prediction with Latent Belief Energy-Based Model
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
• Wasserstein variational gradient descent: From semi-discrete optimal transport to ensemble variational inference
• Imitation-Regularized Offline Learning
• Encoding Distributional Soft Actor-Critic for Autonomous Driving in Multi-lane Scenarios
• Implicit Distributional Reinforcement Learning
• Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy
• Convergent and Efficient Deep Q Network Algorithm
• Identifiability in inverse reinforcement learning
• Multi-Point Bandit Algorithms for Nonstationary Online Nonconvex Optimization
• Stein Self-Repulsive Dynamics: Benefits From Past Samples
• Automata-Guided Hierarchical Reinforcement Learning for Skill Composition
• Learning Graph Structure With A Finite-State Automaton Layer
• Model Based Planning with Energy Based Models
• Kernel Stein Generative Modeling
• Entropy Regularized Motion Planning via Stein Variational Inference
• Ensuring Monotonic Policy Improvement in Entropy-regularized Value-based Reinforcement Learning
• Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Human-guided Robot Behavior Learning: A GAN-assisted Preference-based Reinforcement Learning Approach
• Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning
• Iterative Amortized Policy Optimization
• Implicit Behavioral Cloning
• Multi-Path Policy Optimization
• Reinforcement Learning with Dynamic Boltzmann Softmax Updates
• An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning
• Control What You Can: Intrinsically Motivated Task-Planning Agent
• A Survey on Reproducibility by Evaluating Deep Reinforcement Learning Algorithms on Real-World Robots
• Hierarchical Expert Networks for Meta-Learning
• Learning Representations in Reinforcement Learning:An Information Bottleneck Approach
• Multi-Agent Interactions Modeling with Correlated Policies
• Revisiting Prioritized Experience Replay: A Value Perspective
• Pretrain Soft Q-Learning with Imperfect Demonstrations
• Efficient Learning of Generative Models via Finite-Difference Score Matching
• Optimizing Large-Scale Fleet Management on a Road Network using Multi-Agent Deep Reinforcement Learning with Graph Neural Network
• Inverse reinforcement learning for autonomous navigation via differentiable semantic mapping and planning
• Reinforcement Learning in Non-Markov Market-Making
• Adversarial recovery of agent rewards from latent spaces of the limit order book
• Maximum Entropy Diverse Exploration: Disentangling Maximum Entropy Reinforcement Learning
• Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization
• Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning
• Divergence-Regularized Multi-Agent Actor-Critic
• Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning
• Sampling with Mirrored Stein Operators
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• Temporal-Difference Value Estimation via Uncertainty-Guided Soft Updates
• Energy-based Surprise Minimization for Multi-Agent Value Factorization
• Zeroth-Order Supervised Policy Improvement
• Exploration by Maximizing Rényi Entropy for Reward-Free RL Framework
• A Relation Analysis of Markov Decision Process Frameworks
• Soft Hierarchical Graph Recurrent Networks for Many-Agent Partially Observable Environments
• Advanced Policies: A First-Principles Path from Policy Gradient to Q-Learning
• PFPN: Continuous Control of Physically Simulated Characters using Particle Filtering Policy Network
• Ground States of Quantum Many Body Lattice Models via Reinforcement Learning
• Action Redundancy in Reinforcement Learning
• Neural-to-Tree Policy Distillation with Policy Improvement Criterion
• Bridging Explicit and Implicit Deep Generative Models via Neural Stein Estimators
• Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Problems by Reinforcement Learning
• ISL: A novel approach for deep exploration
• IQ-Learn: Inverse soft-Q Learning for Imitation
• Potential-Based Advice for Stochastic Policy Learning
• Learning Policies through Quantile Regression
• On the Convergence of Approximate and Regularized Policy Iteration Schemes
• Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics
• A Microscopic Pandemic Simulator for Pandemic Prediction Using Scalable Million-Agent Reinforcement Learning
• Maximum entropy exploration in contextual bandits with neural networks and energy based models
• Discrete Action On-Policy Learning with Action-Value Critic
• Task Transfer by Preference-Based Cost Learning
• Learning based E2E Energy Efficient in Joint Radio and NFV Resource Allocation for 5G and Beyond Networks
• Deep PQR: Solving Inverse Reinforcement Learning using Anchor Actions
• Versatile Inverse Reinforcement Learning via Cumulative Rewards
• Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learning
• Context-aware Active Multi-Step Reinforcement Learning
• Homotopy Based Reinforcement Learning with Maximum Entropy for Autonomous Air Combat
• Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures
• Cautious Actor-Critic
• Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning
• Policy-Based Reinforcement Learning for Assortative Matching in Human Behavior Modeling
• Multi-Preference Actor Critic
• CoachNet: An Adversarial Sampling Approach for Reinforcement Learning
• A Joint Reinforcement-Learning Enabled Caching and Cross-Layer Network Code for Sum-Rate Maximization in F-RAN with D2D Communications
• TradeR: Practical Deep Hierarchical Reinforcement Learning for Trade Execution
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning
• Imitation with Neural Density Models
• Adaptive Experience Selection for Policy Gradient
• Geometric Value Iteration: Dynamic Error-Aware KL Regularization for Reinforcement Learning
• Reinforcement Learning for Robust Missile Autopilot Design
• GFlowNet Foundations
• Model Embedding Model-Based Reinforcement Learning
• Leveraging the Variance of Return Sequences for Exploration Policy
• Influence Diagram Bandits: Variational Thompson Sampling for Structured Bandit Problems
• Contrastive Representation Learning with Trainable Augmentation Channel
• A Quadratic Actor Network for Model-Free Reinforcement Learning
• Planning and Learning Using Adaptive Entropy Tree Search
• Learn to Talk via Proactive Knowledge Transfer
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization
• Exploring More When It Needs in Deep Reinforcement Learning