Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning

Abstract

Deep reinforcement learning (RL) is an optimization-driven framework for producing control strategies for general dynamical systems without explicit reliance on process models. Good results have been reported in simulation. Here we demonstrate the challenges in implementing a state of the art deep RL algorithm on a real physical system. Aspects include the interplay between software and existing hardware; experiment design and sample efficiency; training subject to input constraints; and interpretability of the algorithm and control law. At the core of our approach is the use of a PID controller as the trainable RL policy. In addition to its simplicity, this approach has several appealing features: No additional hardware needs to be added to the control system, since a PID controller can easily be implemented through a standard programmable logic controller; the control law can easily be initialized in a "safe'' region of the parameter space; and the final product -- a well-tuned PID controller -- has a form that practitioners can reason about and deploy with confidence.

arXiv comment37 pages; pre-print

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Learning Quadrupedal Locomotion over Challenging Terrain
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Deep Reinforcement Learning for Process Control: A Primer for Beginners
• Deep Reinforcement Learning in Parameterized Action Space
• Optimal PID and Antiwindup Control Design as a Reinforcement Learning Problem
• A Meta-Reinforcement Learning Approach to Process Control

Cited by in corpus (8)

• Machine learning for industrial sensing and control: A survey and practical perspective
• Generative AI and Process Systems Engineering: The Next Frontier
• Meta-Reinforcement Learning for the Tuning of PI Controllers: An Offline Approach
• Neural Operators for PDE Backstepping Control of First-Order Hyperbolic PIDE with Recycle and Delay
• Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior
• A modular framework for stabilizing deep reinforcement learning control
• Reinforcement Learning-based Control via Y-wise Affine Neural Networks (YANNs)
• Atoms to Processes: The Role of Artificial Intelligence and Machine Learning in Chemical Engineering