Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference

Abstract

Modern deep neural networks increasingly make use of features such as dynamic control flow, data structures and dynamic tensor shapes. Existing deep learning systems focus on optimizing and executing static neural networks which assume a pre-determined model architecture and input data shapes--assumptions which are violated by dynamic neural networks. Therefore, executing dynamic models with deep learning systems is currently both inflexible and sub-optimal, if not impossible. Optimizing dynamic neural networks is more challenging than static neural networks; optimizations must consider all possible execution paths and tensor shapes. This paper proposes Nimble, a high-performance and flexible system to optimize, compile, and execute dynamic neural networks on multiple platforms. Nimble handles model dynamism by introducing a dynamic type system, a set of dynamism-oriented optimizations, and a light-weight virtual machine runtime. Our evaluation demonstrates that Nimble outperforms state-of-the-art deep learning frameworks and runtime systems for dynamic neural networks by up to 20x on hardware platforms including Intel CPUs, ARM CPUs, and Nvidia GPUs.

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Sequence to Sequence Learning with Neural Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• cuDNN: Efficient Primitives for Deep Learning
• DyNet: The Dynamic Neural Network Toolkit
• Composing graphical models with neural networks for structured representations and fast inference
• Learning to Optimize Tensor Programs
• Glow: Graph Lowering Compiler Techniques for Neural Networks
• MLIR: A Compiler Infrastructure for the End of Moore's Law
• Deep Learning with Dynamic Computation Graphs
• Ansor: Generating High-Performance Tensor Programs for Deep Learning
• Optimizing CNN Model Inference on CPUs
• Improving the Expressiveness of Deep Learning Frameworks with Recursion
• JANUS: Fast and Flexible Deep Learning via Symbolic Graph Execution of Imperative Programs

Cited by in corpus (6)

• Relax: Composable Abstractions for End-to-End Dynamic Machine Learning
• Cortex: A Compiler for Recursive Deep Learning Models
• Bring Your Own Codegen to Deep Learning Compiler
• SoD${}^2$: Statically Optimizing Dynamic Deep Neural Network
• The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Padding
• Automatic Generation of Interpolants for Lattice Samplings: Part II -- Implementation and Code Generation