Multitasking Inhibits Semantic Drift

Abstract

When intelligent agents communicate to accomplish shared goals, how do these goals shape the agents' language? We study the dynamics of learning in latent language policies (LLPs), in which instructor agents generate natural-language subgoal descriptions and executor agents map these descriptions to low-level actions. LLPs can solve challenging long-horizon reinforcement learning problems and provide a rich model for studying task-oriented language use. But previous work has found that LLP training is prone to semantic drift (use of messages in ways inconsistent with their original natural language meanings). Here, we demonstrate theoretically and empirically that multitask training is an effective counter to this problem: we prove that multitask training eliminates semantic drift in a well-studied family of signaling games, and show that multitask training of neural LLPs in a complex strategy game reduces drift and while improving sample efficiency.

arXiv commentNAACL 2021

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Sequence Level Training with Recurrent Neural Networks
• Deep Reinforcement Learning for Dialogue Generation
• Modular Multitask Reinforcement Learning with Policy Sketches
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Language as an Abstraction for Hierarchical Deep Reinforcement Learning
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Sample Complexity of Multi-task Reinforcement Learning
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning
• Communicative need modulates competition in language change