Multi-Task Reinforcement Learning with Soft Modularization

Abstract

Multi-task learning is a very challenging problem in reinforcement learning. While training multiple tasks jointly allow the policies to share parameters across different tasks, the optimization problem becomes non-trivial: It remains unclear what parameters in the network should be reused across tasks, and how the gradients from different tasks may interfere with each other. Thus, instead of naively sharing parameters across tasks, we introduce an explicit modularization technique on policy representation to alleviate this optimization issue. Given a base policy network, we design a routing network which estimates different routing strategies to reconfigure the base network for each task. Instead of directly selecting routes for each task, our task-specific policy uses a method called soft modularization to softly combine all the possible routes, which makes it suitable for sequential tasks. We experiment with various robotics manipulation tasks in simulation and show our method improves both sample efficiency and performance over strong baselines by a large margin.

arXiv commentOur project page: https://rchalyang.github.io/SoftModule

References in corpus (18)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Multi-Task Learning as Multi-Objective Optimization
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Modular Multitask Reinforcement Learning with Policy Sketches
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Gradient Surgery for Multi-Task Learning
• Adapting Auxiliary Losses Using Gradient Similarity
• Routing Networks and the Challenges of Modular and Compositional Computation
• Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
• Learning to Compose Skills
• Learning Dynamic Routing for Semantic Segmentation
• Learning to Navigate Using Mid-Level Visual Priors
• Composing Task-Agnostic Policies with Deep Reinforcement Learning

Cited by in corpus (12)

• Multi-Task Learning with Deep Neural Networks: A Survey
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
• Multi-task Batch Reinforcement Learning with Metric Learning
• T3S: Improving Multi-Task Reinforcement Learning with Task-Specific Feature Selector and Scheduler
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Making