LambdaNetworks: Modeling Long-Range Interactions Without Attention

Abstract

We present lambda layers -- an alternative framework to self-attention -- for capturing long-range interactions between an input and structured contextual information (e.g. a pixel surrounded by other pixels). Lambda layers capture such interactions by transforming available contexts into linear functions, termed lambdas, and applying these linear functions to each input separately. Similar to linear attention, lambda layers bypass expensive attention maps, but in contrast, they model both content and position-based interactions which enables their application to large structured inputs such as images. The resulting neural network architectures, LambdaNetworks, significantly outperform their convolutional and attentional counterparts on ImageNet classification, COCO object detection and COCO instance segmentation, while being more computationally efficient. Additionally, we design LambdaResNets, a family of hybrid architectures across different scales, that considerably improves the speed-accuracy tradeoff of image classification models. LambdaResNets reach excellent accuracies on ImageNet while being 3.2 - 4.4x faster than the popular EfficientNets on modern machine learning accelerators. When training with an additional 130M pseudo-labeled images, LambdaResNets achieve up to a 9.5x speed-up over the corresponding EfficientNet checkpoints.

Accepted for publication at the International Conference in Learning Representations 2021 (Spotlight)

References in corpus (20)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Learning Transferable Visual Models From Natural Language Supervision
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• Longformer: The Long-Document Transformer
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Generating Long Sequences with Sparse Transformers
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Axial Attention in Multidimensional Transformers
• Reformer: The Efficient Transformer
• Deep Networks with Stochastic Depth
• Stand-Alone Self-Attention in Vision Models
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Rethinking Attention with Performers
• Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
• Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network
• Fast Transformer Decoding: One Write-Head is All You Need
• A Cheap Linear Attention Mechanism with Fast Lookups and Fixed-Size Representations
• Video-based Person Re-identification via 3D Convolutional Networks and Non-local Attention

Cited by in corpus (25)

• Transformers in Vision: A Survey
• EfficientNetV2: Smaller Models and Faster Training
• High-Performance Large-Scale Image Recognition Without Normalization
• UNETR: Transformers for 3D Medical Image Segmentation
• Revisiting ResNets: Improved Training and Scaling Strategies
• RegionViT: Regional-to-Local Attention for Vision Transformers
• Rethinking Spatial Dimensions of Vision Transformers
• An Attention Free Transformer
• Dynamic Neural Networks: A Survey
• Pay Attention to MLPs
• Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
• Involution: Inverting the Inherence of Convolution for Visual Recognition
• Relative Positional Encoding for Transformers with Linear Complexity
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Softmax-free Linear Transformers
• Scene Transformer: A unified architecture for predicting multiple agent trajectories
• FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
• One Self-Configurable Model to Solve Many Abstract Visual Reasoning Problems
• Task-Adaptive Neural Network Search with Meta-Contrastive Learning
• KVT: k-NN Attention for Boosting Vision Transformers
• LambdaUNet: 2.5D Stroke Lesion Segmentation of Diffusion-weighted MR Images
• Unlocking Pixels for Reinforcement Learning via Implicit Attention
• PointMixer: MLP-Mixer for Point Cloud Understanding
• Perception Framework through Real-Time Semantic Segmentation and Scene Recognition on a Wearable System for the Visually Impaired