Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots

Abstract

We introduce the Self-Adaptive Goal Generation - Robust Intelligent Adaptive Curiosity (SAGG-RIAC) architecture as an intrinsi- cally motivated goal exploration mechanism which allows active learning of inverse models in high-dimensional redundant robots. This allows a robot to efficiently and actively learn distributions of parameterized motor skills/policies that solve a corresponding distribution of parameterized tasks/goals. The architecture makes the robot sample actively novel parameterized tasks in the task space, based on a measure of competence progress, each of which triggers low-level goal-directed learning of the motor policy pa- rameters that allow to solve it. For both learning and generalization, the system leverages regression techniques which allow to infer the motor policy parameters corresponding to a given novel parameterized task, and based on the previously learnt correspondences between policy and task parameters. We present experiments with high-dimensional continuous sensorimotor spaces in three different robotic setups: 1) learning the inverse kinematics in a highly-redundant robotic arm, 2) learning omnidirectional locomotion with motor primitives in a quadruped robot, 3) an arm learning to control a fishing rod with a flexible wire. We show that 1) exploration in the task space can be a lot faster than exploration in the actuator space for learning inverse models in redundant robots; 2) selecting goals maximizing competence progress creates developmental trajectories driving the robot to progressively focus on tasks of increasing complexity and is statistically significantly more efficient than selecting tasks randomly, as well as more efficient than different standard active motor babbling methods; 3) this architecture allows the robot to actively discover which parts of its task space it can learn to reach and which part it cannot.

References in corpus (4)

• Interactive Policy Learning through Confidence-Based Autonomy
• Path Integral Policy Improvement with Covariance Matrix Adaptation
• Body Schema Acquisition through Active Learning
• Bootstrapping Intrinsically Motivated Learning with Human Demonstrations

Cited by in corpus (74)

• Visual Reinforcement Learning with Imagined Goals
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Episodic Curiosity through Reachability
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• Reverse Curriculum Generation for Reinforcement Learning
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
• Reset-free Trial-and-Error Learning for Robot Damage Recovery
• Multi-Armed Bandits for Intelligent Tutoring Systems
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• Curriculum Learning for Speech Emotion Recognition from Crowdsourced Labels
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• A Laplacian Framework for Option Discovery in Reinforcement Learning
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Active choice of teachers, learning strategies and goals for a socially guided intrinsic motivation learner
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Scaling MAP-Elites to Deep Neuroevolution
• Socially Guided Intrinsic Motivation for Robot Learning of Motor Skills
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Automatic Curriculum Learning through Value Disagreement
• Intrinsic Motivation and Episodic Memories for Robot Exploration of High-Dimensional Sensory Spaces
• Automated curricula through setter-solver interactions
• From exploration to control: learning object manipulation skills through novelty search and local adaptation
• Computational Theories of Curiosity-Driven Learning
• A novel plasticity rule can explain the development of sensorimotor intelligence
• Improved Learning of Robot Manipulation Tasks via Tactile Intrinsic Motivation
• Accuracy-based Curriculum Learning in Deep Reinforcement Learning
• Learning agent's spatial configuration from sensorimotor invariants
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments
• Reward-Respecting Subtasks for Model-Based Reinforcement Learning
• Behavior Self-Organization Supports Task Inference for Continual Robot Learning
• Autonomous development and learning in artificial intelligence and robotics: Scaling up deep learning to human--like learning
• Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems
• Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems
• Hyperparameter Auto-tuning in Self-Supervised Robotic Learning
• Autonomous discovery of the goal space to learn a parameterized skill
• Learning Inverse Statics Models Efficiently
• Where do goals come from? A Generic Approach to Autonomous Goal-System Development
• Robots Learn Increasingly Complex Tasks with Intrinsic Motivation and Automatic Curriculum Learning
• Learning a Set of Interrelated Tasks by Using Sequences of Motor Policies for a Strategic Intrinsically Motivated Learner
• Exploring Exploration: Comparing Children with RL Agents in Unified Environments
• Active Learning for Autonomous Intelligent Agents: Exploration, Curiosity, and Interaction
• Parametric-Task MAP-Elites
• HyperNCA: Growing Developmental Networks with Neural Cellular Automata
• DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
• Continuous Value Iteration (CVI) Reinforcement Learning and Imaginary Experience Replay (IER) for learning multi-goal, continuous action and state space controllers
• Self-supervised Visual Reinforcement Learning with Object-centric Representations
• Exploration via Hindsight Goal Generation
• Intrinsically Motivated Acquisition of Modular Slow Features for Humanoids in Continuous and Non-Stationary Environments
• Control What You Can: Intrinsically Motivated Task-Planning Agent
• Open challenges in understanding development and evolution of speech forms: The roles of embodied self-organization, motivation and active exploration
• Adversarial Active Exploration for Inverse Dynamics Model Learning
• Self-organized control for musculoskeletal robots
• Temporal Abstraction in Reinforcement Learning with the Successor Representation
• Using Curiosity for an Even Representation of Tasks in Continual Offline Reinforcement Learning
• Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks
• Emergent Jaw Predominance in Vocal Development through Stochastic Optimization
• Exploration of Self-Propelling Droplets Using a Curiosity Driven Robotic Assistant
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals
• Learning in Sparse Rewards settings through Quality-Diversity algorithms
• Human-Machine Collaborative Optimization via Apprenticeship Scheduling
• Online Body Schema Adaptation through Cost-Sensitive Active Learning
• Autonomous Open-Ended Learning of Interdependent Tasks
• Autonomous Goal Exploration using Learned Goal Spaces for Visuomotor Skill Acquisition in Robots
• Local Online Motor Babbling: Learning Motor Abundance of A Musculoskeletal Robot Arm
• A solution to generalized learning from small training sets found in infants repeated visual experiences of individual objects
• Being curious about the answers to questions: novelty search with learned attention
• What Can This Robot Do? Learning from Appearance and Experiments
• Autonomous learning of multiple, context-dependent tasks