A reinforcement learning approach to rare trajectory sampling

Abstract

Very often when studying non-equilibrium systems one is interested in analysing dynamical behaviour that occurs with very low probability, so called rare events. In practice, since rare events are by definition atypical, they are often difficult to access in a statistically significant way. What are required are strategies to "make rare events typical" so that they can be generated on demand. Here we present such a general approach to adaptively construct a dynamics that efficiently samples atypical events. We do so by exploiting the methods of reinforcement learning (RL), which refers to the set of machine learning techniques aimed at finding the optimal behaviour to maximise a reward associated with the dynamics. We consider the general perspective of dynamical trajectory ensembles, whereby rare events are described in terms of ensemble reweighting. By minimising the distance between a reweighted ensemble and that of a suitably parametrised controlled dynamics we arrive at a set of methods similar to those of RL to numerically approximate the optimal dynamics that realises the rare behaviour of interest. As simple illustrations we consider in detail the problem of excursions of a random walker, for the case of rare events with a finite time horizon; and the problem of a studying current statistics of a particle hopping in a ring geometry, for the case of an infinite time horizon. We discuss natural extensions of the ideas presented here, including to continuous-time Markov systems, first passage time problems and non-Markovian dynamics.

arXiv comment55+6 pages, 7+1 figures

References in corpus (72)

• Continuous control with deep reinforcement learning
• Overcoming catastrophic forgetting in neural networks
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• The large deviation approach to statistical mechanics
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Reinforcement Learning with Deep Energy-Based Policies
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Direct evaluation of large-deviation functions
• Reinforcement Learning in Different Phases of Quantum Control
• Optimal control as a graphical model inference problem
• Large deviations and ensembles of trajectories in stochastic models
• Reinforcement Learning with Neural Networks for Quantum Feedback
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Sample Efficient Actor-Critic with Experience Replay
• Simulating rare events in dynamical processes
• Aspects of non-equilibrium in classical and quantum systems: slow relaxation and glasses, dynamical large deviations, quantum non-ergodicity, and open quantum dynamics
• A numerical approach to large deviations in continuous-time
• Variational and optimal control representations of conditioned and driven processes
• Ergodicity and large deviations in physical systems with stochastic dynamics
• Population dynamics method with a multi-canonical feedback control
• Effective interactions and large deviations in stochastic processes
• Optimizing active work: Dynamical phase transitions, collective motion, and jamming
• A unified view of entropy-regularized Markov decision processes
• A Theory of Regularized Markov Decision Processes
• Adaptive importance sampling for control and inference
• Finite-Size Scaling of a First-Order Dynamical Phase Transition: Adaptive Population Dynamics and an Effective Model
• Using matrix product states to study the dynamical large deviations of kinetically constrained models
• Reinforcement learning for autonomous preparation of Floquet-engineered states: Inverting the quantum Kapitza oscillator
• DMRG-study of current and activity fluctuations near non-equilibrium phase transitions
• Exact fluctuations of nonequilibrium steady states from approximate auxiliary dynamics
• Large deviations conditioned on large deviations I: Markov chain and Langevin equation
• Rare behavior of growth processes via umbrella sampling of trajectories
• True Online Temporal-Difference Learning
• Off-Policy Actor-Critic
• Importance sampling large deviations in nonequilibrium steady states. I
• Adaptive sampling of large deviations
• Reinforcement Learning for Digital Quantum Simulation
• Computation of Large Deviation Statistics via Iterative Measurement-and-Feedback Procedure
• Large deviations conditioned on large deviations II: Fluctuating hydrodynamics
• Dynamical phase behavior of the single- and multi-lane asymmetric simple exclusion process via matrix product states
• Classical stochastic dynamics and continuous matrix product states: gauge transformations, conditioned and driven processes, and equivalence of trajectory ensembles
• Variational control forces for enhanced sampling of nonequilibrium molecular dynamics simulations
• Dynamical Phase Transitions in a 2D Classical Nonequilibrium Model via 2D Tensor Networks
• Policy Gradient based Quantum Approximate Optimization Algorithm
• Expected Policy Gradients for Reinforcement Learning
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Extreme Spin Squeezing from Deep Reinforcement Learning
• Discounted Reinforcement Learning Is Not an Optimization Problem
• Phases of quantum dimers from ensembles of classical stochastic trajectories
• Direct evaluation of dynamical large-deviation rate functions using a variational ansatz
• Evolutionary reinforcement learning of dynamical large deviations
• Reinforcement learning for semi-autonomous approximate quantum eigensolver
• Dynamical large deviations of two-dimensional kinetically constrained models using a neural-network state ansatz
• Large deviations and optimal control forces for hard particles in one dimension
• Policy Guided Monte Carlo: Reinforcement Learning Markov Chain Dynamics
• Malliavin weight sampling: a practical guide
• Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation
• Mean Actor Critic
• Entanglement statistics in Markovian open quantum systems: a matter of mutation and selection
• A Deep Learning Functional Estimator of Optimal Dynamics for Sampling Large Deviations
• Multi-step Reinforcement Learning: A Unifying Algorithm
• Effective Multi-step Temporal-Difference Learning for Non-Linear Function Approximation
• Constructing Auxiliary Dynamics for Nonequilibrium Stationary States by Variance Minimization
• Reconciling $\lambda$-Returns with Experience Replay
• General non-linear Bellman equations
• Two geometric input transformation methods for fast online reinforcement learning with neural nets
• A Theoretical Connection Between Statistical Physics and Reinforcement Learning
• Overcoming Catastrophic Interference in Online Reinforcement Learning with Dynamic Self-Organizing Maps
• A Tensor Network Approach to Finite Markov Decision Processes

Cited by in corpus (28)

• Phase Diagram of Active Brownian Spheres: Crystallization and the Metastability of Motility-Induced Phase Separation
• Reinforcement Learning for Many-Body Ground-State Preparation Inspired by Counterdiabatic Driving
• Learning nonequilibrium control forces to characterize dynamical phase transitions
• Reinforcement learning of rare diffusive dynamics
• Finite time large deviations via matrix product states
• Optimal sampling of dynamical large deviations via matrix product states
• Direct evaluation of rare events in active matter from variational path sampling
• Efficient and Robust Entanglement Generation with Deep Reinforcement Learning for Quantum Metrology
• Dynamical large deviations of two-dimensional kinetically constrained models using a neural-network state ansatz
• Phase Coexistence Implications of Violating Newton's Third Law
• Landau theory for finite-time dynamical phase transitions
• Generating discrete-time constrained random walks and Lévy flights
• A large deviation theory perspective on nanoscale transport phenomena
• Generalized optimal paths and weight distributions revealed through the large deviations of random walks on networks
• Generating stochastic trajectories with global dynamical constraints
• Generating constrained run-and-tumble trajectories
• Modeling of biomolecular machines in non-equilibrium steady states
• Entropy Regularized Reinforcement Learning Using Large Deviation Theory
• Adaptive power method for estimating large deviations in Markov chains
• Convergence of the Integral Fluctuation Theorem estimator for nonequilibrium Markov systems
• Physics-informed graph neural networks enhance scalability of variational nonequilibrium optimal control
• Variational time reversal for free energy estimation in nonequilibrium steady states
• Transport properties of diffusive particles conditioned to survive in trapping environments
• Towards neural reinforcement learning for large deviations in nonequilibrium systems with memory
• Learn Quasi-stationary Distributions of Finite State Markov Chain
• Self-interacting processes via Doob conditioning
• Making rare events typical in $d$-dimensional chaotic maps
• Mechanical analysis of a dynamical phase transition for particles in a channel