A unified strategy for implementing curiosity and empowerment driven reinforcement learning

Abstract

Although there are many approaches to implement intrinsically motivated artificial agents, the combined usage of multiple intrinsic drives remains still a relatively unexplored research area. Specifically, we hypothesize that a mechanism capable of quantifying and controlling the evolution of the information flow between the agent and the environment could be the fundamental component for implementing a higher degree of autonomy into artificial intelligent agents. This paper propose a unified strategy for implementing two semantically orthogonal intrinsic motivations: curiosity and empowerment. Curiosity reward informs the agent about the relevance of a recent agent action, whereas empowerment is implemented as the opposite information flow from the agent to the environment that quantifies the agent's potential of controlling its own future. We show that an additional homeostatic drive is derived from the curiosity reward, which generalizes and enhances the information gain of a classical curious/heterostatic reinforcement learning agent. We show how a shared internal model by curiosity and empowerment facilitates a more efficient training of the empowerment function. Finally, we discuss future directions for further leveraging the interplay between these two intrinsic rewards.

arXiv comment13 pages, 8 figures

References in corpus (4)

• Continuous control with deep reinforcement learning
• Variational Intrinsic Control
• A Unified Bellman Equation for Causal Information and Value in Markov Decision Processes
• Information Theoretically Aided Reinforcement Learning for Embodied Agents

Cited by in corpus (9)

• A survey on intrinsic motivation in reinforcement learning
• Reinforcement Learning through Active Inference
• Action and Perception as Divergence Minimization
• A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
• AvE: Assistance via Empowerment
• Automated and Autonomous Experiment in Electron and Scanning Probe Microscopy
• Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communication
• Explore and Control with Adversarial Surprise
• Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments