TVM: An Automated End-to-End Optimizing Compiler for Deep Learning

Abstract

There is an increasing need to bring machine learning to a wide diversity of hardware devices. Current frameworks rely on vendor-specific operator libraries and optimize for a narrow range of server-class GPUs. Deploying workloads to new platforms -- such as mobile phones, embedded devices, and accelerators (e.g., FPGAs, ASICs) -- requires significant manual effort. We propose TVM, a compiler that exposes graph-level and operator-level optimizations to provide performance portability to deep learning workloads across diverse hardware back-ends. TVM solves optimization challenges specific to deep learning, such as high-level operator fusion, mapping to arbitrary hardware primitives, and memory latency hiding. It also automates optimization of low-level programs to hardware characteristics by employing a novel, learning-based cost modeling method for rapid exploration of code optimizations. Experimental results show that TVM delivers performance across hardware back-ends that are competitive with state-of-the-art, hand-tuned libraries for low-power CPU, mobile GPU, and server-class GPUs. We also demonstrate TVM's ability to target new accelerator back-ends, such as the FPGA-based generic deep learning accelerator. The system is open sourced and in production use inside several major companies.

Significantly improved version, add automated optimization

References in corpus (11)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Recurrent Neural Network Regularization
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Theano: new features and speed improvements
• Identity Mappings in Deep Residual Networks
• Fast Algorithms for Convolutional Neural Networks
• Loo.py: transformation-based code generation for GPUs and CPUs
• DLVM: A modern compiler infrastructure for deep learning systems
• High performance ultra-low-precision convolutions on mobile devices

Cited by in corpus (57)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• RetinaFace: Single-stage Dense Face Localisation in the Wild
• MCUNet: Tiny Deep Learning on IoT Devices
• Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems
• Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
• High-Throughput CNN Inference on Embedded ARM big.LITTLE Multi-Core Processors
• Relay: A New IR for Machine Learning Frameworks
• Bias in Machine Learning Models Can Be Significantly Mitigated by Careful Training: Evidence from Neuroimaging Studies
• Kernel Operations on the GPU, with Autodiff, without Memory Overflows
• Optimizing CNN Model Inference on CPUs
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• Fusion-Catalyzed Pruning for Optimizing Deep Learning on Intelligent Edge Devices
• GSPMD: General and Scalable Parallelization for ML Computation Graphs
• VarGNet: Variable Group Convolutional Neural Network for Efficient Embedded Computing
• Cloudy with high chance of DBMS: A 10-year prediction for Enterprise-Grade ML
• HAWQV3: Dyadic Neural Network Quantization
• I-BERT: Integer-only BERT Quantization
• STONNE: A Detailed Architectural Simulator for Flexible Neural Network Accelerators
• Parity Models: A General Framework for Coding-Based Resilience in ML Inference
• BLK-REW: A Unified Block-based DNN Pruning Framework using Reweighted Regularization Method
• Real-Time Execution of Large-scale Language Models on Mobile
• Cortex: A Compiler for Recursive Deep Learning Models
• Intelligent-Unrolling: Exploiting Regular Patterns in Irregular Applications
• Face Detection with Feature Pyramids and Landmarks
• DNNVM : End-to-End Compiler Leveraging Heterogeneous Optimizations on FPGA-based CNN Accelerators
• Bridging Control-Centric and Data-Centric Optimization
• SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
• Automating Generation of Low Precision Deep Learning Operators
• Q-EEGNet: an Energy-Efficient 8-bit Quantized Parallel EEGNet Implementation for Edge Motor-Imagery Brain--Machine Interfaces
• ProTuner: Tuning Programs with Monte Carlo Tree Search
• SESAME: Software defined Enclaves to Secure Inference Accelerators with Multi-tenant Execution
• FusionStitching: Deep Fusion and Code Generation for Tensorflow Computations on GPUs
• The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Padding
• The OoO VLIW JIT Compiler for GPU Inference
• Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
• The landscape of software for tensor computations
• Energy-Aware DNN Graph Optimization
• SparseDNN: Fast Sparse Deep Learning Inference on CPUs
• Simulating Execution Time of Tensor Programs using Graph Neural Networks
• RLgraph: Modular Computation Graphs for Deep Reinforcement Learning
• AI Powered Compiler Techniques for DL Code Optimization
• Long Document Ranking with Query-Directed Sparse Transformer
• Accelerating Multi-Model Inference by Merging DNNs of Different Weights
• Machine Learning Systems for Intelligent Services in the IoT: A Survey
• DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distribution
• Enabling Cross-Domain Communication: How to Bridge the Gap between AI and HW Engineers
• SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning
• ISA Mapper: A Compute and Hardware Agnostic Deep Learning Compiler
• Using Python for Model Inference in Deep Learning
• Understanding Cache Boundness of ML Operators on ARM Processors
• Improving DNN Fault Tolerance using Weight Pruning and Differential Crossbar Mapping for ReRAM-based Edge AI
• Automatic Generation of Fast and Accurate Performance Models for Deep Neural Network Accelerators
• Layout Design for Intelligent Warehouse by Evolution with Fitness Approximation