Stripe: Tensor Compilation via the Nested Polyhedral Model

Abstract

Hardware architectures and machine learning (ML) libraries evolve rapidly. Traditional compilers often fail to generate high-performance code across the spectrum of new hardware offerings. To mitigate, engineers develop hand-tuned kernels for each ML library update and hardware upgrade. Unfortunately, this approach requires excessive engineering effort to scale or maintain with any degree of state-of-the-art performance. Here we present a Nested Polyhedral Model for representing highly parallelizable computations with limited dependencies between iterations. This model provides an underlying framework for an intermediate representation (IR) called Stripe, amenable to standard compiler techniques while naturally modeling key aspects of modern ML computing. Stripe represents parallelism, efficient memory layout, and multiple compute units at a level of abstraction amenable to automatic optimization. We describe how Stripe enables a compiler for ML in the style of LLVM that allows independent development of algorithms, optimizations, and hardware accelerators. We also discuss the design exploration advantages of Stripe over kernel libraries and schedule-based or schedule-space-based code generation.

References in corpus (4)

• Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning
• Relay: A New IR for Machine Learning Frameworks
• Optimizing CNN Model Inference on CPUs
• DLVM: A modern compiler infrastructure for deep learning systems

Cited by in corpus (8)

• MLIR: A Compiler Infrastructure for the End of Moore's Law
• Efficient Execution of Quantized Deep Learning Models: A Compiler Approach
• Automatic Kernel Generation for Volta Tensor Cores
• High-Performance Deep Learning via a Single Building Block
• The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Padding
• Compiling Neural Networks for a Computational Memory Accelerator
• Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloads
• Comparing the costs of abstraction for DL frameworks