Measuring Compositionality in Representation Learning

Abstract

Many machine learning algorithms represent input data with vector embeddings or discrete codes. When inputs exhibit compositional structure (e.g. objects built from parts or procedures from subroutines), it is natural to ask whether this compositional structure is reflected in the the inputs' learned representations. While the assessment of compositionality in languages has received significant attention in linguistics and adjacent fields, the machine learning literature lacks general-purpose tools for producing graded measurements of compositional structure in more general (e.g. vector-valued) representation spaces. We describe a procedure for evaluating compositionality by measuring how well the true representation-producing model can be approximated by a model that explicitly composes a collection of inferred representational primitives. We use the procedure to provide formal and empirical characterizations of compositional structure in a variety of settings, exploring the relationship between compositionality and learning dynamics, human judgments, representational similarity, and generalization.

ICLR 2019

References in corpus (7)

• Distributed Representations of Words and Phrases and their Compositionality
• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Opening the Black Box of Deep Neural Networks via Information
• Multi-Agent Cooperation and the Emergence of (Natural) Language
• Visual Reference Resolution using Attention Memory for Visual Dialog
• Emergence of Communication in an Interactive World with Consistent Speakers
• Towards Multi-Agent Communication-Based Language Learning

Cited by in corpus (14)

• Zero-shot Node Classification with Decomposed Graph Prototype Network
• Compositional Languages Emerge in a Neural Iterated Learning Model
• Learning by Abstraction: The Neural State Machine
• Compositionality decomposed: how do neural networks generalise?
• Emergent Language: A Survey and Taxonomy
• Compositional Visual Generation and Inference with Energy Based Models
• Multimodal Generative Models for Compositional Representation Learning
• Inductive Bias and Language Expressivity in Emergent Communication
• On the Interplay Between Fine-tuning and Composition in Transformers
• Emergence of Numeric Concepts in Multi-Agent Autonomous Communication
• EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use case
• Structural Inductive Biases in Emergent Communication
• Compositional Embeddings for Multi-Label One-Shot Learning
• CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning