Intelligent problem-solving as integrated hierarchical reinforcement learning
arXiv:2208.08731 · doi:10.1038/s42256-021-00433-9
Abstract
According to cognitive psychology and related disciplines, the development of complex problem-solving behaviour in biological agents depends on hierarchical cognitive mechanisms. Hierarchical reinforcement learning is a promising computational approach that may eventually yield comparable problem-solving behaviour in artificial agents and robots. However, to date the problem-solving abilities of many human and non-human animals are clearly superior to those of artificial systems. Here, we propose steps to integrate biologically inspired hierarchical mechanisms to enable advanced problem-solving skills in artificial agents. Therefore, we first review the literature in cognitive psychology to highlight the importance of compositional abstraction and predictive processing. Then we relate the gained insights with contemporary hierarchical reinforcement learning methods. Interestingly, our results suggest that all identified cognitive mechanisms have been implemented individually in isolated computational architectures, raising the question of why there exists no single unifying architecture that integrates them. As our final contribution, we address this question by providing an integrative perspective on the computational challenges to develop such a unifying architecture. We expect our results to guide the development of more sophisticated cognitively inspired hierarchical machine learning architectures.
arXiv commentPublished as accepted article in Nature Machine Intelligence: https://www.nature.com/articles/s42256-021-00433-9. arXiv admin note: substantial text overlap with arXiv:2012.10147
References in corpus (13)
- A Brief Survey of Deep Reinforcement Learning
- SHOP2: An HTN Planning System
- Learning by Playing - Solving Sparse Reward Tasks from Scratch
- FeUdal Networks for Hierarchical Reinforcement Learning
- Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
- Learning and Transfer of Modulated Locomotor Controllers
- Intrinsic Motivation and Episodic Memories for Robot Exploration of High-Dimensional Sensory Spaces
- Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
- Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imagination
- On the role of planning in model-based deep reinforcement learning
- Composing Task-Agnostic Policies with Deep Reinforcement Learning
- Hierarchical Reinforcement Learning with Abductive Planning
- Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL