Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions

Abstract

Deep learning models with convolutional and recurrent networks are now ubiquitous and analyze massive amounts of audio, image, video, text and graph data, with applications in automatic translation, speech-to-text, scene understanding, ranking user preferences, ad placement, etc. Competing frameworks for building these networks such as TensorFlow, Chainer, CNTK, Torch/PyTorch, Caffe1/2, MXNet and Theano, explore different tradeoffs between usability and expressiveness, research or production orientation and supported hardware. They operate on a DAG of computational operators, wrapping high-performance libraries such as CUDNN (for NVIDIA GPUs) or NNPACK (for various CPUs), and automate memory allocation, synchronization, distribution. Custom operators are needed where the computation does not fit existing high-performance library calls, usually at a high engineering cost. This is frequently required when new operators are invented by researchers: such operators suffer a severe performance penalty, which limits the pace of innovation. Furthermore, even if there is an existing runtime call these frameworks can use, it often doesn't offer optimal performance for a user's particular network architecture and dataset, missing optimizations between operators as well as optimizations that can be done knowing the size and shape of data. Our contributions include (1) a language close to the mathematics of deep learning called Tensor Comprehensions, (2) a polyhedral Just-In-Time compiler to convert a mathematical description of a deep learning DAG into a CUDA kernel with delegated memory management and synchronization, also providing optimizations such as operator fusion and specialization for specific sizes, (3) a compilation cache populated by an autotuner. [Abstract cutoff]

References in corpus (16)

• TensorFlow: A system for large-scale machine learning
• Improving neural networks by preventing co-adaptation of feature detectors
• Practical Bayesian Optimization of Machine Learning Algorithms
• Deep Residual Learning for Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• A Review of Relational Machine Learning for Knowledge Graphs
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
• Aggregated Residual Transformations for Deep Neural Networks
• Wide & Deep Learning for Recommender Systems
• Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• Active Libraries: Rethinking the roles of compilers and libraries
• Fast Algorithms for Convolutional Neural Networks
• Loo.py: transformation-based code generation for GPUs and CPUs

Cited by in corpus (74)

• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Glow: Graph Lowering Compiler Techniques for Neural Networks
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• Relay: A New IR for Machine Learning Frameworks
• Mind Mappings: Enabling Efficient Algorithm-Accelerator Mapping Space Search
• Ansor: Generating High-Performance Tensor Programs for Deep Learning
• Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
• Optimizing CNN Model Inference on CPUs
• Compiler Support for Sparse Tensor Computations in MLIR
• Neural Kernels Without Tangents
• DISTAL: The Distributed Tensor Algebra Compiler
• UNIT: Unifying Tensorized Instruction Compilation
• Dynamic Space-Time Scheduling for GPU Inference
• Pure Tensor Program Rewriting via Access Patterns (Representation Pearl)
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• HAWQV3: Dyadic Neural Network Quantization
• A model-driven approach for a new generation of adaptive libraries
• Chameleon: Adaptive Code Optimization for Expedited Deep Neural Network Compilation
• DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusion
• Reinforcement Learning and Adaptive Sampling for Optimized DNN Compilation
• Automatic Kernel Generation for Volta Tensor Cores
• TIRAMISU: A Polyhedral Compiler for Dense and Sparse Deep Learning
• ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learning
• Cortex: A Compiler for Recursive Deep Learning Models
• Intelligent-Unrolling: Exploiting Regular Patterns in Irregular Applications
• PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
• DNNVM : End-to-End Compiler Leveraging Heterogeneous Optimizations on FPGA-based CNN Accelerators
• SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
• Efficient Contraction of Large Tensor Networks for Weighted Model Counting through Graph Decompositions
• Neural Architecture Search using Property Guided Synthesis
• A Proposal for Loop-Transformation Pragmas
• Bridging Control-Centric and Data-Centric Optimization
• High-Performance Deep Learning via a Single Building Block
• Bring Your Own Codegen to Deep Learning Compiler
• ProTuner: Tuning Programs with Monte Carlo Tree Search
• From Domain-Specific Languages to Memory-Optimized Accelerators for Fluid Dynamics
• Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
• Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devices
• A Highly Configurable Hardware/Software Stack for DNN Inference Acceleration
• Optimizing DNN Compilation for Distributed Training with Joint OP and Tensor Fusion
• Array Languages Make Neural Networks Fast
• The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Padding
• Differentiating a Tensor Language
• CoSA: Scheduling by Constrained Optimization for Spatial Accelerators
• Sentinel: Runtime Data Management on Heterogeneous Main MemorySystems for Deep Learning
• DISC: A Dynamic Shape Compiler for Machine Learning Workloads
• On the Representation of Partially Specified Implementations and its Application to the Optimization of Linear Algebra Kernels on GPU
• The OoO VLIW JIT Compiler for GPU Inference
• Sparse Tensor Algebra Optimizations with Workspaces
• Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
• MetaTune: Meta-Learning Based Cost Model for Fast and Efficient Auto-tuning Frameworks
• HASCO: Towards Agile HArdware and Software CO-design for Tensor Computation
• Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloads
• Tensor Relational Algebra for Machine Learning System Design
• Tuna: A Static Analysis Approach to Optimizing Deep Neural Networks
• PolyDL: Polyhedral Optimizations for Creation of High Performance DL primitives
• A Unified Iteration Space Transformation Framework for Sparse and Dense Tensor Algebra
• Comparing the costs of abstraction for DL frameworks
• A Reinforcement Learning Environment for Polyhedral Optimizations
• Fireiron: A Scheduling Language for High-Performance Linear Algebra on GPUs
• DLFusion: An Auto-Tuning Compiler for Layer Fusion on Deep Neural Network Accelerator
• Pushing the boundaries of parallel Deep Learning -- A practical approach
• Spatial Sharing of GPU for Autotuning DNN models
• POLO: a POLicy-based Optimization library
• Automatic Creation of High-Bandwidth Memory Architectures from Domain-Specific Languages: The Case of Computational Fluid Dynamics
• CoolerSpace: A Language for Physically Correct and Computationally Efficient Color Programming
• Maximum Flows in Parametric Graph Templates
• Ricci-Notation Tensor Framework for Model-based Approaches to Imaging
• ISA Mapper: A Compute and Hardware Agnostic Deep Learning Compiler
• Symbolical Index Reduction and Completion Rules for Importing Tensor Index Notation into Programming Languages
• Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations
• miniKanren as a Tool for Symbolic Computation in Python