Optimizing CNN Model Inference on CPUs

Abstract

The popularity of Convolutional Neural Network (CNN) models and the ubiquity of CPUs imply that better performance of CNN model inference on CPUs can deliver significant gain to a large number of users. To improve the performance of CNN inference on CPUs, current approaches like MXNet and Intel OpenVINO usually treat the model as a graph and use the high-performance libraries such as Intel MKL-DNN to implement the operations of the graph. While achieving reasonable performance on individual operations from the off-the-shelf libraries, this solution makes it inflexible to conduct optimizations at the graph level, as the local operation-level optimizations are predefined. Therefore, it is restrictive and misses the opportunity to optimize the end-to-end inference pipeline as a whole. This paper presents \emph{NeoCPU}, a comprehensive approach of CNN model inference on CPUs that employs a full-stack and systematic scheme of optimizations. \emph{NeoCPU} optimizes the operations as templates without relying on third-parties libraries, which enables further improvement of the performance via operation- and graph-level joint optimization. Experiments show that \emph{NeoCPU} achieves up to 3.45$\times$ lower latency for CNN model inference than the current state-of-the-art implementations on various kinds of popular CPUs.

15 pages, 4 figures, published at USENIX ATC '19

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: A system for large-scale machine learning
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• cuDNN: Efficient Primitives for Deep Learning
• Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
• Learning to Optimize Tensor Programs
• Glow: Graph Lowering Compiler Techniques for Neural Networks
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning
• ZNN - A Fast and Scalable Algorithm for Training 3D Convolutional Networks on Multi-Core and Many-Core Shared Memory Machines
• FFT Convolutions are Faster than Winograd on Modern CPUs, Here is Why
• Scheduling Computation Graphs of Deep Learning Models on Manycore CPUs

Cited by in corpus (13)

• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• High-Throughput CNN Inference on Embedded ARM big.LITTLE Multi-Core Processors
• Ansor: Generating High-Performance Tensor Programs for Deep Learning
• UNIT: Unifying Tensorized Instruction Compilation
• Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference
• Stripe: Tensor Compilation via the Nested Polyhedral Model
• DNNVM : End-to-End Compiler Leveraging Heterogeneous Optimizations on FPGA-based CNN Accelerators
• Bring Your Own Codegen to Deep Learning Compiler
• High-Performance Deep Learning via a Single Building Block
• A Highly Configurable Hardware/Software Stack for DNN Inference Acceleration
• Sentinel: Runtime Data Management on Heterogeneous Main MemorySystems for Deep Learning
• SparseDNN: Fast Sparse Deep Learning Inference on CPUs