Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning

Abstract

We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To this end, we introduce a general-purpose deep learning architecture that takes as input the entire dataset instead of processing one datapoint at a time. Our approach uses self-attention to reason about relationships between datapoints explicitly, which can be seen as realizing non-parametric models using parametric attention mechanisms. However, unlike conventional non-parametric models, we let the model learn end-to-end from the data how to make use of other datapoints for prediction. Empirically, our models solve cross-datapoint lookup and complex reasoning tasks unsolvable by traditional deep learning models. We show highly competitive results on tabular data, early results on CIFAR-10, and give insight into how the model makes use of the interactions between points.

arXiv commentAccepted for publication at NeurIPS 2021. First two authors contributed equally

References in corpus (17)

• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MissForest - nonparametric missing value imputation for mixed-type data
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Generating Long Sequences with Sparse Transformers
• Lookahead Optimizer: k steps forward, 1 step back
• Axial Attention in Multidimensional Transformers
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
• Image Transformer
• Perceiver: General Perception with Iterative Attention
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control
• LieTransformer: Equivariant self-attention for Lie Groups
• ImageNet-21K Pretraining for the Masses

Cited by in corpus (3)

• Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
• Tabular Machine Learning Methods for Predicting Gas Turbine Emissions
• Beyond Individual Input for Deep Anomaly Detection on Tabular Data