A2: Extracting Cyclic Switchings from DOB-nets for Rejecting Excessive Disturbances

Abstract

Reinforcement Learning (RL) is limited in practice by its gray-box nature, which is responsible for insufficient trustiness from users, unsatisfied interpretation for human intervention, inadequate analysis for future improvement, etc. This paper seeks to partially characterize the interplay between dynamical environments and the DOB-net. The DOB-net obtained from RL solves a set of Partially Observable Markovian Decision Processes (POMDPs). The transition function of each POMDP is largely determined by the environments, which are excessive external disturbances in this research. This paper proposes an Attention-based Abstraction (A${}^2$) approach to extract a finite-state automaton, referred to as a Key Moore Machine Network (KMMN), to capture the switching mechanisms exhibited by the DOB-net in dealing with multiple such POMDPs. This approach first quantizes the controlled platform by learning continuous-discrete interfaces. Then it extracts the KMMN by finding the key hidden states and transitions that attract sufficient attention from the DOB-net. Within the resultant KMMN, this study found three patterns of cyclic switchings (between key hidden states), showing controls near their saturation are synchronized with unknown disturbances. Interestingly, the found switching mechanism has appeared previously in the design of hybrid control for often-saturated systems. It is further interpreted via an analogy to the discrete-event subsystem in the hybrid control.

References in corpus (14)

• Continuous control with deep reinforcement learning
• Neural Architecture Search with Reinforcement Learning
• Trust Region Policy Optimization
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models
• Visualizing and Understanding Recurrent Networks
• Continuous Deep Q-Learning with Model-based Acceleration
• Control of Memory, Active Perception, and Action in Minecraft
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Learning Finite State Representations of Recurrent Policy Networks
• Particle Filter Networks with Application to Visual Localization
• DOB-Net: Actively Rejecting Unknown Excessive Time-Varying Disturbances