Reinforcement learning of rare diffusive dynamics

Abstract

We present a method to probe rare molecular dynamics trajectories directly using reinforcement learning. We consider trajectories that are conditioned to transition between regions of configuration space in finite time, like those relevant in the study of reactive events, as well as trajectories exhibiting rare fluctuations of time-integrated quantities in the long time limit, like those relevant in the calculation of large deviation functions. In both cases, reinforcement learning techniques are used to optimize an added force that minimizes the Kullback-Leibler divergence between the conditioned trajectory ensemble and a driven one. Under the optimized added force, the system evolves the rare fluctuation as a typical one, affording a variational estimate of its likelihood in the original trajectory ensemble. Low variance gradients employing value functions are proposed to increase the convergence of the optimal force. The method we develop employing these gradients leads to efficient and accurate estimates of both the optimal force and the likelihood of the rare event for a variety of model systems.

arXiv comment23 pages, 8 figures

References in corpus (75)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• The Entropy Production Fluctuation Theorem and the Nonequilibrium Work Relation for Free Energy Differences
• The large deviation approach to statistical mechanics
• Soft Actor-Critic Algorithms and Applications
• A Gallavotti-Cohen Type Symmetry in the Large Deviation Functional for Stochastic Dynamics
• String Method for the Study of Rare Events
• Non equilibrium steady states: fluctuations and large deviations of the density and of the current
• Solving Rubik's Cube with a Robot Hand
• Current fluctuations in non-equilibrium diffusive systems: an additivity principle
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Direct evaluation of large-deviation functions
• Nonequilibrium Markov processes conditioned on large deviations
• Reinforcement Learning in Different Phases of Quantum Control
• Optimal control as a graphical model inference problem
• Universal bounds on current fluctuations
• Large deviations and ensembles of trajectories in stochastic models
• Reinforcement Learning with Neural Networks for Quantum Feedback
• A Unified Approach to Enhanced Sampling
• A numerical approach to large deviations in continuous-time
• Variational and optimal control representations of conditioned and driven processes
• Population dynamics method with a multi-canonical feedback control
• Effective interactions and large deviations in stochastic processes
• Current large deviations for driven periodic diffusions
• A unified view of entropy-regularized Markov decision processes
• Optimizing active work: Dynamical phase transitions, collective motion, and jamming
• Making rare events typical in Markovian open quantum systems
• A Theory of Regularized Markov Decision Processes
• Adaptive importance sampling for control and inference
• Onsager-Machlup theory for nonequilibrium steady states and fluctuation theorems
• Global optimization of quantum dynamics with AlphaZero deep exploration
• Finite-Size Scaling of a First-Order Dynamical Phase Transition: Adaptive Population Dynamics and an Effective Model
• Computing Committor Functions for the Study of Rare Events Using Deep Learning
• Reinforcement learning for autonomous preparation of Floquet-engineered states: Inverting the quantum Kapitza oscillator
• Exact fluctuations of nonequilibrium steady states from approximate auxiliary dynamics
• Path Integral Control and State Dependent Feedback
• Entropy production fluctuations encode collective behavior in active matter
• How dissipation constrains fluctuations in nonequilibrium liquids: Diffusion, structure and biased interactions
• Transition path time distributions
• Importance sampling large deviations in nonequilibrium steady states. I
• Large Deviation Function for a Driven Underdamped Particle in a Periodic Potential
• Practical rare event sampling for extreme mesoscale weather
• A reinforcement learning approach to rare trajectory sampling
• Adaptive sampling of large deviations
• Fluctuating observation time ensembles in the thermodynamics of trajectories
• Precision shooting: Sampling long transition pathways
• Collective motion in large deviations of active particles
• Computing return times or return periods with rare event algorithms
• Current fluctuations of interacting active Brownian particles
• Deep Reinforcement Learning of Transition States
• Variational control forces for enhanced sampling of nonequilibrium molecular dynamics simulations
• Dissipation bounds the amplification of transition rates far from equilibrium
• Reinforcement learning of rare diffusive dynamics
• Steered Transition Path Sampling
• Policy Gradient based Quantum Approximate Optimization Algorithm
• Nonlinear transport coefficients from large deviation functions
• Onsager-Machlup action-based path sampling and its combination with replica exchange for diffusive and multiple pathways
• Extreme Spin Squeezing from Deep Reinforcement Learning
• Transition Path Theory from Biased Simulations
• Optimal sampling of dynamical large deviations via matrix product states
• Evolutionary reinforcement learning of dynamical large deviations
• Preserving Correlations Between Trajectories for Efficient Path Sampling
• Variational design principles for nonequilibrium colloidal assembly
• Non-intersecting Brownian bridges in the flat-to-flat geometry
• Large deviations and optimal control forces for hard particles in one dimension
• Transport Coefficients from Large Deviation Functions
• Methods to locate Saddle Points in Complex Landscapes
• Stochastic Optimal Control as Non-equilibrium Statistical Mechanics: Calculus of Variations over Density and Current
• Generating discrete-time constrained random walks and Lévy flights
• Malliavin weight sampling: a practical guide
• Active Importance Sampling for Variational Objectives Dominated by Rare Events: Consequences for Optimization and Generalization
• A large deviation theory perspective on nanoscale transport phenomena
• A Deep Learning Functional Estimator of Optimal Dynamics for Sampling Large Deviations
• Path sampling with Stochastic Dynamics: Some new Algorithms
• Inverse Design of Non-Equilibrium Steady-States: A Large Deviation Approach
• Ground States of Quantum Many Body Lattice Models via Reinforcement Learning

Cited by in corpus (10)

• Learning nonequilibrium control forces to characterize dynamical phase transitions
• Reinforcement learning of rare diffusive dynamics
• Perspective: How to overcome dynamical density functional theory
• Finite time large deviations via matrix product states
• Direct evaluation of rare events in active matter from variational path sampling
• Shortcuts in stochastic systems and control of biophysical processes
• Generating stochastic trajectories with global dynamical constraints
• Entropy Regularized Reinforcement Learning Using Large Deviation Theory
• Adaptive power method for estimating large deviations in Markov chains
• Physics-informed graph neural networks enhance scalability of variational nonequilibrium optimal control