Communication-Optimal Convolutional Neural Nets

Abstract

Efficiently executing convolutional neural nets (CNNs) is important in many machine-learning tasks. Since the cost of moving a word of data, either between levels of a memory hierarchy or between processors over a network, is much higher than the cost of an arithmetic operation, minimizing data movement is critical to performance optimization. In this paper, we present both new lower bounds on data movement needed for CNNs, and optimal sequential algorithms that attain these lower bounds. In most common cases, our optimal algorithms can attain significantly more data reuse than matrix multiplication.

References in corpus (2)

• Minimizing Communication in Linear Algebra
• Parallelepipeds obtaining HBL lower bounds

Cited by in corpus (8)

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Mesh-TensorFlow: Deep Learning for Supercomputers
• Flexible Communication Avoiding Matrix Multiplication on FPGA with High-Level Synthesis
• Derivation and Analysis of Fast Bilinear Algorithms for Convolution
• I/O Lower Bounds for Auto-tuning of Convolutions in CNNs
• Communication lower bounds for nested bilinear algorithms via rank expansion of Kronecker products
• ISA Mapper: A Compute and Hardware Agnostic Deep Learning Compiler
• Pebbles, Graphs, and a Pinch of Combinatorics: Towards Tight I/O Lower Bounds for Statically Analyzable Programs