Perceiver: General Perception with Iterative Attention

Abstract

Biological systems perceive the world by simultaneously processing high-dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perception models used in deep learning on the other hand are designed for individual modalities, often relying on domain-specific assumptions such as the local grid structures exploited by virtually all existing vision models. These priors introduce helpful inductive biases, but also lock models to individual modalities. In this paper we introduce the Perceiver - a model that builds upon Transformers and hence makes few architectural assumptions about the relationship between its inputs, but that also scales to hundreds of thousands of inputs, like ConvNets. The model leverages an asymmetric attention mechanism to iteratively distill inputs into a tight latent bottleneck, allowing it to scale to handle very large inputs. We show that this architecture is competitive with or outperforms strong, specialized models on classification tasks across various modalities: images, point clouds, audio, video, and video+audio. The Perceiver obtains performance comparable to ResNet-50 and ViT on ImageNet without 2D convolutions by directly attending to 50,000 pixels. It is also competitive in all modalities in AudioSet.

arXiv commentICML 2021

References in corpus (21)

• Geometric deep learning: going beyond Euclidean data
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Gaussian Error Linear Units (GELUs)
• Convolutional Sequence to Sequence Learning
• Longformer: The Long-Document Transformer
• Scaling Laws for Neural Language Models
• Linformer: Self-Attention with Linear Complexity
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Group Equivariant Convolutional Networks
• Generating Long Sequences with Sparse Transformers
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• Axial Attention in Multidimensional Transformers
• Reformer: The Efficient Transformer
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• High-Performance Large-Scale Image Recognition Without Normalization
• High-Performance Neural Networks for Visual Object Classification
• Audiovisual SlowFast Networks for Video Recognition
• Random Feature Attention
• Multi-level Attention Model for Weakly Supervised Audio Classification
• BP-Transformer: Modelling Long-Range Context via Binary Partitioning
• Coordination Among Neural Modules Through a Shared Global Workspace

Cited by in corpus (26)

• Transformers in Vision: A Survey
• Attention Bottlenecks for Multimodal Fusion
• XCiT: Cross-Covariance Image Transformers
• Efficient Training of Audio Transformers with Patchout
• Woodpecker: Hallucination Correction for Multimodal Large Language Models
• SOFT: Softmax-free Transformer with Linear Complexity
• Long-Short Transformer: Efficient Transformers for Language and Vision
• Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning
• Inductive Biases and Variable Creation in Self-Attention Mechanisms
• Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning
• Softmax-free Linear Transformers
• Transformers predicting the future. Applying attention in next-frame and time series forecasting
• Long Short-Term Transformer for Online Action Detection
• The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning
• Proposal-based Few-shot Sound Event Detection for Speech and Environmental Sounds with Perceivers
• Towards mental time travel: a hierarchical memory for reinforcement learning agents
• Hierarchically Compositional Tasks and Deep Convolutional Networks
• Improved Transformer for High-Resolution GANs
• Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks
• PolyViT: Co-training Vision Transformers on Images, Videos and Audio
• $\infty$-former: Infinite Memory Transformer
• The Uncanny Similarity of Recurrence and Depth
• User-Oriented Smart General AI System under Causal Inference
• Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition
• Shape registration in the time of transformers
• Multi-Glimpse Network: A Robust and Efficient Classification Architecture based on Recurrent Downsampled Attention