Analytical Characterization and Design Space Exploration for Optimization of CNNs

Abstract

Moving data through the memory hierarchy is a fundamental bottleneck that can limit the performance of core algorithms of machine learning, such as convolutional neural networks (CNNs). Loop-level optimization, including loop tiling and loop permutation, are fundamental transformations to reduce data movement. However, the search space for finding the best loop-level optimization configuration is explosively large. This paper develops an analytical modeling approach for finding the best loop-level optimization configuration for CNNs on multi-core CPUs. Experimental evaluation shows that this approach achieves comparable or better performance than state-of-the-art libraries and auto-tuning based optimizers for CNNs.

In proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '21), April 19-23, 2021, Virtual, USA

References in corpus (3)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• LIKWID: Lightweight Performance Tools
• A Fast Analytical Model of Fully Associative Caches

Cited by in corpus (6)

• A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators
• A Multi-level Compiler Backend for Accelerated Micro-kernels Targeting RISC-V ISA Extensions
• Full Version: (De/Re)-Composition of Data-Parallel Computations via Multi-Dimensional Homomorphisms
• Efficient distributed algorithms for Convolutional Neural Networks
• A Unified Framework for Automated Code Transformation and Pragma Insertion
• Introducing Instruction-Accurate Simulators for Performance Estimation of Autotuning Workloads