Episodic Memory for Learning Subjective-Timescale Models

Abstract

In model-based learning, an agent's model is commonly defined over transitions between consecutive states of an environment even though planning often requires reasoning over multi-step timescales, with intermediate states either unnecessary, or worse, accumulating prediction error. In contrast, intelligent behaviour in biological organisms is characterised by the ability to plan over varying temporal scales depending on the context. Inspired by the recent works on human time perception, we devise a novel approach to learning a transition dynamics model, based on the sequences of episodic memories that define the agent's subjective timescale - over which it learns world dynamics and over which future planning is performed. We implement this in the framework of active inference and demonstrate that the resulting subjective-timescale model (STM) can systematically vary the temporal extent of its predictions while preserving the same computational efficiency. Additionally, we show that STM predictions are more likely to introduce future salient events (for example new objects coming into view), incentivising exploration of new areas of the environment. As a result, STM produces more informative action-conditioned roll-outs that assist the agent in making better decisions. We validate significant improvement in our STM agent's performance in the Animal-AI environment against a baseline system, trained using the environment's objective-timescale dynamics.

References in corpus (14)

• Prioritized Experience Replay
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Recurrent World Models Facilitate Policy Evolution
• Model-Based Reinforcement Learning for Atari
• A free energy principle for a particular physics
• Imagination-Augmented Agents for Deep Reinforcement Learning
• Control of Memory, Active Perception, and Action in Minecraft
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Deep Active Inference
• Bayesian policy selection using active inference
• Prediction and Control with Temporal Segment Models
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• Fast deep reinforcement learning using online adjustments from the past
• Learning What to Remember: Long-term Episodic Memory Networks for Learning from Streaming Data