Countering Language Drift with Seeded Iterated Learning
arXiv:2003.12694
Abstract
Pretraining on human corpus and then finetuning in a simulator has become a standard pipeline for training a goal-oriented dialogue agent. Nevertheless, as soon as the agents are finetuned to maximize task completion, they suffer from the so-called language drift phenomenon: they slowly lose syntactic and semantic properties of language as they only focus on solving the task. In this paper, we propose a generic approach to counter language drift called Seeded iterated learning (SIL). We periodically refine a pretrained student agent by imitating data sampled from a newly generated teacher agent. At each time step, the teacher is created by copying the student agent, before being finetuned to maximize task completion. SIL does not require external syntactic constraint nor semantic knowledge, making it a valuable task-agnostic finetuning protocol. We evaluate SIL in a toy-setting Lewis Game, and then scale it up to the translation game with natural language. In both settings, SIL helps counter language drift as well as it improves the task completion compared to baselines.
References in corpus (11)
- Natural Language Processing (almost) from Scratch
- Categorical Reparameterization with Gumbel-Softmax
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Towards a Human-like Open-Domain Chatbot
- Multi-Agent Cooperation and the Emergence of (Natural) Language
- Dialogue Learning With Human-In-The-Loop
- Evaluating Visual Conversational Agents via Cooperative Human-AI Games
- Emergence of Compositional Language with Deep Generational Transmission
- Compositional Languages Emerge in a Neural Iterated Learning Model
- The Emergence of Compositional Languages for Numeric Concepts Through Iterated Learning in Neural Agents
- Interactive Reinforcement Learning for Object Grounding via Self-Talking