Entropy Regularized Reinforcement Learning Using Large Deviation Theory

Abstract

Reinforcement learning (RL) is an important field of research in machine learning that is increasingly being applied to complex optimization problems in physics. In parallel, concepts from physics have contributed to important advances in RL with developments such as entropy-regularized RL. While these developments have led to advances in both fields, obtaining analytical solutions for optimization in entropy-regularized RL is currently an open problem. In this paper, we establish a mapping between entropy-regularized RL and research in non-equilibrium statistical mechanics focusing on Markovian processes conditioned on rare events. In the long-time limit, we apply approaches from large deviation theory to derive exact analytical results for the optimal policy and optimal dynamics in Markov Decision Process (MDP) models of reinforcement learning. The results obtained lead to a novel analytical and computational framework for entropy-regularized RL which is validated by simulations. The mapping established in this work connects current research in reinforcement learning and non-equilibrium statistical mechanics, thereby opening new avenues for the application of analytical and computational approaches from one field to cutting-edge problems in the other.

References in corpus (25)

• A high-bias, low-variance introduction to Machine Learning for physicists
• Why does deep and cheap learning work so well?
• First-order dynamical phase transition in models of glasses: an approach based on ensembles of histories
• Direct evaluation of large-deviation functions
• Nonequilibrium Markov processes conditioned on large deviations
• Optimal control as a graphical model inference problem
• Large deviations and ensembles of trajectories in stochastic models
• Nonequilibrium microcanonical and canonical ensembles and their equivalence
• Quasi-stationary distributions and population processes
• Machine learning meets quantum physics
• Variational and optimal control representations of conditioned and driven processes
• Population dynamics method with a multi-canonical feedback control
• Construction of a Coordinate Bethe Ansatz for the asymmetric simple exclusion process with open boundaries
• Rules for transition rates in nonequilibrium steady states
• Topological Quantum Compiling with Reinforcement Learning
• Measurement Based Feedback Quantum Control With Deep Reinforcement Learning for Double-well Non-linear Potential
• Detailed balance has a counterpart in non-equilibrium steady states
• A reinforcement learning approach to rare trajectory sampling
• Reinforcement Learning for Digital Quantum Simulation
• Adaptive Optics control using Model-Based Reinforcement Learning
• Reinforcement learning of rare diffusive dynamics
• Properties of a non-equilibrium heat bath
• Evolutionary reinforcement learning of dynamical large deviations
• Absence of dissipation in trajectory ensembles biased by currents
• Physics-informed graph neural networks enhance scalability of variational nonequilibrium optimal control