Emergent Translation in Multi-Agent Communication

Abstract

While most machine translation systems to date are trained on large parallel corpora, humans learn language in a different way: by being grounded in an environment and interacting with other humans. In this work, we propose a communication game where two agents, native speakers of their own respective languages, jointly learn to solve a visual referential task. We find that the ability to understand and translate a foreign language emerges as a means to achieve shared goals. The emergent translation is interactive and multimodal, and crucially does not require parallel corpora, but only monolingual, independent text and corresponding images. Our proposed translation model achieves this by grounding the source and target languages into a shared visual modality, and outperforms several baselines on both word-level and sentence-level translation tasks. Furthermore, we show that agents in a multilingual community learn to translate better and faster than in a bilingual communication setting.

Accepted to ICLR 2018

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Deep Residual Learning for Image Recognition
• Categorical Reparameterization with Gumbel-Softmax
• Convolutional Sequence to Sequence Learning
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Learning Multiagent Communication with Backpropagation
• Imagination improves Multimodal Translation
• Multimodal Pivots for Image Caption Translation
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Order-Embeddings of Images and Language
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
• Multimodal Attention for Neural Machine Translation
• Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence
• Emergent Communication in a Multi-Modal, Multi-Step Referential Game
• The University of Edinburgh's Neural MT Systems for WMT17
• STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset
• Multilingual Multi-modal Embeddings for Natural Language Processing
• Translating Neuralese

Cited by in corpus (23)

• Unsupervised Machine Translation Using Monolingual Corpora Only
• Anti-efficient encoding in emergent communication
• Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks
• Emergence of Compositional Language with Deep Generational Transmission
• Meta-Learning for Low-Resource Neural Machine Translation
• Ease-of-Teaching and Language Structure from Emergent Communication
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
• Emergent Communication Pretraining for Few-Shot Machine Translation
• How agents see things: On visual representations in an emergent language game
• Measuring non-trivial compositionality in emergent communication
• Improving Visually Grounded Sentence Representations with Self-Attention
• Incorporating Pragmatic Reasoning Communication into Emergent Language
• Avoiding hashing and encouraging visual semantics in referential emergent language games
• Rat big, cat eaten! Ideas for a useful deep-agent protolanguage
• Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication
• Symbolic Semantic Segmentation and Interpretation of COVID-19 Lung Infections in Chest CT volumes based on Emergent Languages
• A Survey on Low-Resource Neural Machine Translation
• Vehicle Communication Strategies for Simulated Highway Driving
• Expressivity of Emergent Language is a Trade-off between Contextual Complexity and Unpredictability
• The emergence of visual semantics through communication games
• Cooperative image captioning
• Bootstrapping Disjoint Datasets for Multilingual Multimodal Representation Learning
• PatchGame: Learning to Signal Mid-level Patches in Referential Games