General Value Function Networks

Abstract

State construction is important for learning in partially observable environments. A general purpose strategy for state construction is to learn the state update using a Recurrent Neural Network (RNN), which updates the internal state using the current internal state and the most recent observation. This internal state provides a summary of the observed sequence, to facilitate accurate predictions and decision-making. At the same time, specifying and training RNNs is notoriously tricky, particularly as the common strategy to approximate gradients back in time, called truncated Back-prop Through Time (BPTT), can be sensitive to the truncation window. Further, domain-expertise--which can usually help constrain the function class and so improve trainability--can be difficult to incorporate into complex recurrent units used within RNNs. In this work, we explore how to use multi-step predictions to constrain the RNN and incorporate prior knowledge. In particular, we revisit the idea of using predictions to construct state and ask: does constraining (parts of) the state to consist of predictions about the future improve RNN trainability? We formulate a novel RNN architecture, called a General Value Function Network (GVFN), where each internal state component corresponds to a prediction about the future represented as a value function. We first provide an objective for optimizing GVFNs, and derive several algorithms to optimize this objective. We then show that GVFNs are more robust to the truncation level, in many cases only requiring one-step gradient updates.

arXiv commentPublished in the Journal of Artificial Intelligence Research

References in corpus (25)

• LSTM: A Search Space Odyssey
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• On the difficulty of training Recurrent Neural Networks
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Deep Recurrent Q-Learning for Partially Observable MDPs
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Measuring Catastrophic Forgetting in Neural Networks
• Learning Longer-term Dependencies in RNNs with Auxiliary Losses
• Temporal-Difference Networks
• Discovery of Useful Questions as Auxiliary Tasks
• Approximating Real-Time Recurrent Learning with Random Kronecker Factors
• Learning from Conditional Distributions via Dual Embeddings
• Predictive Linear-Gaussian Models of Stochastic Dynamical Systems
• Unifying task specification in reinforcement learning
• Predictive State Recurrent Neural Networks
• Learning Causal Models Online
• Online Off-policy Prediction
• Learning to Filter with Predictive State Inference Machines
• Learning to Predict Independent of Span
• Predictive-State Decoders: Encoding the Future into Recurrent Networks
• Learning predictive representations in autonomous driving to improve deep reinforcement learning
• Optimal Kronecker-Sum Approximation of Real Time Recurrent Learning
• Directly Estimating the Variance of the λ-Return Using Temporal-Difference Methods
• Temporal-Difference Networks for Dynamical Systems with Continuous Observations and Actions

Cited by in corpus (7)

• Contrastive Explanations for Reinforcement Learning via Embedded Self Predictions
• Using Echo State Networks to Approximate Value Functions for Control
• Composite Q-learning: Multi-scale Q-function Decomposition and Separable Optimization
• Affordance as general value function: A computational model
• Predictive Representation Learning for Language Modeling
• Finding Useful Predictions by Meta-gradient Descent to Improve Decision-making
• Learning State Representations from Random Deep Action-conditional Predictions