Hierarchical principles of embodied reinforcement learning: A review

Abstract

Cognitive Psychology and related disciplines have identified several critical mechanisms that enable intelligent biological agents to learn to solve complex problems. There exists pressing evidence that the cognitive mechanisms that enable problem-solving skills in these species build on hierarchical mental representations. Among the most promising computational approaches to provide comparable learning-based problem-solving abilities for artificial agents and robots is hierarchical reinforcement learning. However, so far the existing computational approaches have not been able to equip artificial agents with problem-solving abilities that are comparable to intelligent animals, including human and non-human primates, crows, or octopuses. Here, we first survey the literature in Cognitive Psychology, and related disciplines, and find that many important mental mechanisms involve compositional abstraction, curiosity, and forward models. We then relate these insights with contemporary hierarchical reinforcement learning methods, and identify the key machine intelligence approaches that realise these mechanisms. As our main result, we show that all important cognitive mechanisms have been implemented independently in isolated computational architectures, and there is simply a lack of approaches that integrate them appropriately. We expect our results to guide the development of more sophisticated cognitively inspired hierarchical methods, so that future artificial agents achieve a problem-solving performance on the level of intelligent animals.

References in corpus (24)

• A Brief Survey of Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• SHOP2: An HTN Planning System
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Deep Reinforcement Learning: An Overview
• Recurrent World Models Facilitate Policy Evolution
• State Representation Learning for Control: An Overview
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• FeUdal Networks for Hierarchical Reinforcement Learning
• Playing FPS Games with Deep Reinforcement Learning
• Learning and Transfer of Modulated Locomotor Controllers
• Language as an Abstraction for Hierarchical Deep Reinforcement Learning
• A Laplacian Framework for Option Discovery in Reinforcement Learning
• Learning Multi-Level Hierarchies with Hindsight
• State Abstraction in MAXQ Hierarchical Reinforcement Learning
• Intrinsic Motivation and Episodic Memories for Robot Exploration of High-Dimensional Sensory Spaces
• Computational Theories of Curiosity-Driven Learning
• Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies
• The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously
• Classifying Options for Deep Reinforcement Learning
• Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imagination
• Composing Task-Agnostic Policies with Deep Reinforcement Learning
• Hierarchical Reinforcement Learning with Abductive Planning
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL