Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL
arXiv:2012.09737
Abstract
Reinforcement learning holds tremendous promise in accelerator controls. The primary goal of this paper is to show how this approach can be utilised on an operational level on accelerator physics problems. Despite the success of model-free reinforcement learning in several domains, sample-efficiency still is a bottle-neck, which might be encompassed by model-based methods. We compare well-suited purely model-based to model-free reinforcement learning applied to the intensity optimisation on the FERMI FEL system. We find that the model-based approach demonstrates higher representational power and sample-efficiency, while the asymptotic performance of the model-free method is slightly superior. The model-based algorithm is implemented in a DYNA-style using an uncertainty aware model, and the model-free algorithm is based on tailored deep Q-learning. In both cases, the algorithms were implemented in a way, which presents increased noise robustness as omnipresent in accelerator control problems. Code is released in https://github.com/MathPhysSim/FERMI_RL_Paper.
arXiv comment13 pages, 17 figures - minor changes and adaption to physical review journals
References in corpus (16)
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic Algorithms and Applications
- Emergence of Locomotion Behaviours in Rich Environments
- Gaussian Processes for Data-Efficient Learning in Robotics and Control
- Distributed Distributional Deterministic Policy Gradients
- Model-Ensemble Trust-Region Policy Optimization
- Challenges of Real-World Reinforcement Learning
- Benchmarking Model-Based Reinforcement Learning
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
- When to Trust Your Model: Model-Based Policy Optimization
- Exploring Model-based Planning with Policy Networks
- Multi-Objective Bayesian Optimization for Accelerator Tuning
- Uncertainty in Neural Networks: Approximately Bayesian Ensembling
- Real-time Artificial Intelligence for Accelerator Control: A Study at the Fermilab Booster
- Autonomous Control of a Particle Accelerator using Deep Reinforcement Learning