Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks

Abstract

Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant. We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set. The model consists of an encoder and a decoder, both of which rely on attention mechanisms. In an effort to reduce computational complexity, we introduce an attention scheme inspired by inducing point methods from sparse Gaussian process literature. It reduces the computation time of self-attention from quadratic to linear in the number of elements in the set. We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating the state-of-the-art performance compared to recent methods for set-structured data.

ICML 2019

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Theoretical Models of Learning to Learn
• Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction
• Neural Episodic Control
• Learning from Distributions via Support Measure Machines
• Probabilistic symmetries and invariant neural networks

Cited by in corpus (22)

• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• Amortized Variational Inference: A Systematic Review
• An end-to-end attention-based approach for learning on graphs
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation
• Ball Trajectory Inference from Multi-Agent Sports Contexts Using Set Transformer and Hierarchical Bi-LSTM
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Few-Shot Learning as Domain Adaptation: Algorithm and Analysis
• Unveiling quantum phase transitions from traps in variational quantum algorithms
• A Mathematical Theory of Attention
• DuMLP-Pin: A Dual-MLP-dot-product Permutation-invariant Network for Set Feature Extraction
• Structured Generative Models for Scene Understanding
• Neural Clustering Processes
• Set-to-Sequence Methods in Machine Learning: a Review
• MxML: Mixture of Meta-Learners for Few-Shot Classification
• Adaptive parameters identification for nonlinear dynamics using deep permutation invariant networks
• Better Set Representations For Relational Reasoning
• Mapping in a cycle: Sinkhorn regularized unsupervised learning for point cloud shapes
• On the Regularity of Attention
• EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications
• MIDAS: Multi-agent Interaction-aware Decision-making with Adaptive Strategies for Urban Autonomous Navigation
• You May Not Need Order in Time Series Forecasting
• Knowledge as Invariance -- History and Perspectives of Knowledge-augmented Machine Learning