Multiscale Deep Equilibrium Models

Abstract

We propose a new class of implicit networks, the multiscale deep equilibrium model (MDEQ), suited to large-scale and highly hierarchical pattern recognition domains. An MDEQ directly solves for and backpropagates through the equilibrium points of multiple feature resolutions simultaneously, using implicit differentiation to avoid storing intermediate states (and thus requiring only $O(1)$ memory consumption). These simultaneously-learned multi-resolution features allow us to train a single model on a diverse set of tasks and loss functions, such as using a single MDEQ to perform both image classification and semantic segmentation. We illustrate the effectiveness of this approach on two large-scale vision tasks: ImageNet classification and semantic segmentation on high-resolution images from the Cityscapes dataset. In both settings, MDEQs are able to match or exceed the performance of recent competitive computer vision models: the first time such performance and scale have been achieved by an implicit deep learning approach. The code and pre-trained models are at https://github.com/locuslab/mdeq .

arXiv commentNeurIPS 2020 Oral

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• TensorFlow: A system for large-scale machine learning
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Wide Residual Networks
• Universal Transformers
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• UNet++: A Nested U-Net Architecture for Medical Image Segmentation
• Deep Equilibrium Models
• Trellis Networks for Sequence Modeling
• Scalable Differentiable Physics for Learning and Control
• Structured Knowledge Distillation for Dense Prediction

Cited by in corpus (10)

• Long-term predictions of turbulence by implicit U-Net enhanced Fourier neural operator
• Nonlocal Kernel Network (NKN): a Stable and Resolution-Independent Deep Neural Network
• Vision Transformers with Patch Diversification
• Implicit Feature Pyramid Network for Object Detection
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks
• SITReg: Multi-resolution architecture for symmetric, inverse consistent, and topology preserving image registration
• Deep Implicit Optimization enables Robust Learnable Features for Deformable Image Registration
• Memory-Efficient Differentiable Transformer Architecture Search
• Greedy and Random Broyden's Methods with Explicit Superlinear Convergence Rates in Nonlinear Equations
• Neural Fixed-Point Acceleration for Convex Optimization