Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning

Abstract

The Deep Learning (DL) community sees many novel topologies published each year. Achieving high performance on each new topology remains challenging, as each requires some level of manual effort. This issue is compounded by the proliferation of frameworks and hardware platforms. The current approach, which we call "direct optimization", requires deep changes within each framework to improve the training performance for each hardware backend (CPUs, GPUs, FPGAs, ASICs) and requires $\mathcal{O}(fp)$ effort; where $f$ is the number of frameworks and $p$ is the number of platforms. While optimized kernels for deep-learning primitives are provided via libraries like Intel Math Kernel Library for Deep Neural Networks (MKL-DNN), there are several compiler-inspired ways in which performance can be further optimized. Building on our experience creating neon (a fast deep learning library on GPUs), we developed Intel nGraph, a soon to be open-sourced C++ library to simplify the realization of optimized deep learning performance across frameworks and hardware platforms. Initially-supported frameworks include TensorFlow, MXNet, and Intel neon framework. Initial backends are Intel Architecture CPUs (CPU), the Intel(R) Nervana Neural Network Processor(R) (NNP), and NVIDIA GPUs. Currently supported compiler optimizations include efficient memory management and data layout abstraction. In this paper, we describe our overall architecture and its core components. In the future, we envision extending nGraph API support to a wider range of frameworks, hardware (including FPGAs and ASICs), and compiler optimizations (training versus inference optimizations, multi-node and multi-device scaling via efficient sub-graph partitioning, and HW-specific compounding of operations).

References in corpus (1)

• DLVM: A modern compiler infrastructure for deep learning systems

Cited by in corpus (23)

• Glow: Graph Lowering Compiler Techniques for Neural Networks
• SoK: Fully Homomorphic Encryption Compilers
• Optimizing CNN Model Inference on CPUs
• Don't Unroll Adjoint: Differentiating SSA-Form Programs
• Fusion-Catalyzed Pruning for Optimizing Deep Learning on Intelligent Edge Devices
• Enabling Homomorphically Encrypted Inference for Large DNN Models
• nGraph-HE2: A High-Throughput Framework for Neural Network Inference on Encrypted Data
• Stripe: Tensor Compilation via the Nested Polyhedral Model
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• Quantune: Post-training Quantization of Convolutional Neural Networks using Extreme Gradient Boosting for Fast Deployment
• Efficient Execution of Quantized Deep Learning Models: A Compiler Approach
• nGraph-HE: A Graph Compiler for Deep Learning on Homomorphically Encrypted Data
• CIM-MLC: A Multi-level Compilation Stack for Computing-In-Memory Accelerators
• DNNVM : End-to-End Compiler Leveraging Heterogeneous Optimizations on FPGA-based CNN Accelerators
• Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devices
• Optimizing DNN Compilation for Distributed Training with Joint OP and Tensor Fusion
• Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training
• Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
• FlexSA: Flexible Systolic Array Architecture for Efficient Pruned DNN Model Training
• Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning
• NNStreamer: Efficient and Agile Development of On-Device AI Systems
• Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations
• ISA Mapper: A Compute and Hardware Agnostic Deep Learning Compiler