Analogs of Linguistic Structure in Deep Representations
arXiv:1707.08139
Abstract
We investigate the compositional structure of message vectors computed by a deep network trained on a communication game. By comparing truth-conditional representations of encoder-produced message vectors to human-produced referring expressions, we are able to identify aligned (vector, utterance) pairs with the same meaning. We then search for structured relationships among these aligned pairs to discover simple vector space transformations corresponding to negation, conjunction, and disjunction. Our results suggest that neural representations are capable of spontaneously developing a "syntax" with functional analogues to qualitative properties of natural language.
arXiv commentIn EMNLP 2017
References in corpus (7)
- On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
- Learning to Communicate with Deep Multi-Agent Reinforcement Learning
- Learning Multiagent Communication with Backpropagation
- Question Answering with Subgraph Embeddings
- Towards Multi-Agent Communication-Based Language Learning
- Translating Neuralese
- A Joint Speaker-Listener-Reinforcer Model for Referring Expressions