Using Echo State Networks to Approximate Value Functions for Control
arXiv:2102.06258
Abstract
An Echo State Network (ESN) is a type of single-layer recurrent neural network with randomly-chosen internal weights and a trainable output layer. We prove under mild conditions that a sufficiently large Echo State Network can approximate the value function of a broad class of stochastic and deterministic control problems. Such control problems are generally non-Markovian. We describe how the ESN can form the basis for novel and computationally efficient reinforcement learning algorithms in a non-Markovian framework. We demonstrate this theory with two examples. In the first, we use an ESN to solve a deterministic, partially observed, control problem which is a simple game we call `Bee World'. In the second example, we consider a stochastic control problem inspired by a market making problem in mathematical finance. In both cases we can compare the dynamics of the algorithms with analytic solutions to show that even after only a single reinforcement policy iteration the algorithms arrive at a good policy.
References in corpus (6)
- Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
- Embedding and Approximation Theorems for Echo State Networks
- Echo State Networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems
- Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learning
- General Value Function Networks
- On the Universal Transformation of Data-Driven Models to Control Systems