Perceiver IO: A General Architecture for Structured Inputs & Outputs

Abstract

A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible. Current architectures, however, cannot be applied beyond a small set of stereotyped settings, as they bake in domain & task assumptions or scale poorly to large inputs or outputs. In this work, we propose Perceiver IO, a general-purpose architecture that handles data from arbitrary settings while scaling linearly with the size of inputs and outputs. Our model augments the Perceiver with a flexible querying mechanism that enables outputs of various sizes and semantics, doing away with the need for task-specific architecture engineering. The same architecture achieves strong results on tasks spanning natural language and visual understanding, multi-task and multi-modal reasoning, and StarCraft II. As highlights, Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.

arXiv commentICLR 2022 camera ready. Code: https://dpmd.ai/perceiver-code

References in corpus (23)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Natural Language Processing (almost) from Scratch
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Gaussian Error Linear Units (GELUs)
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Scaling Laws for Neural Language Models
• Zero-Shot Text-to-Image Generation
• Linformer: Self-Attention with Linear Complexity
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• One Model To Learn Them All
• High-Performance Large-Scale Image Recognition Without Normalization
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
• Pretrained Transformers as Universal Computation Engines
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Sim2real transfer learning for 3D human pose estimation: motion to the rescue
• Luna: Linear Unified Nested Attention
• Imitating Interactive Intelligence
• Hash Embeddings for Efficient Word Representations
• Coordination Among Neural Modules Through a Shared Global Workspace

Cited by in corpus (19)

• Transformers in Vision: A Survey
• On the Opportunities and Risks of Foundation Models
• Robotic Vision for Human-Robot Interaction and Collaboration: A Survey and Systematic Review
• CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Vision Transformers Need Registers
• Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie
• Inductive Biases and Variable Creation in Self-Attention Mechanisms
• A graph-based probabilistic geometric deep learning framework with online enforcement of physical constraints to predict the criticality of defects in porous materials
• MM-ALT: A Multimodal Automatic Lyric Transcription System
• Robust Semi-Supervised Learning for Self-learning Open-World Classes
• COPER: Continuous Patient State Perceiver
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• LaRa: Latents and Rays for Multi-Camera Bird's-Eye-View Semantic Segmentation
• SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding
• TLCFuse: Temporal Multi-Modality Fusion Towards Occlusion-Aware Semantic Segmentation-Aided Motion Planning
• The Efficiency Misnomer
• Conditional Object-Centric Learning from Video
• Improving the Perceptual Quality of 2D Animation Interpolation