Dissecting the Graphcore IPU Architecture via Microbenchmarking

Abstract

This report focuses on the architecture and performance of the Intelligence Processing Unit (IPU), a novel, massively parallel platform recently introduced by Graphcore and aimed at Artificial Intelligence/Machine Learning (AI/ML) workloads. We dissect the IPU's performance behavior using microbenchmarks that we crafted for the purpose. We study the IPU's memory organization and performance. We study the latency and bandwidth that the on-chip and off-chip interconnects offer, both in point-to-point transfers and in a spectrum of collective operations, under diverse loads. We evaluate the IPU's compute power over matrix multiplication, convolution, and AI/ML primitives. We discuss actual performance in comparison with its theoretical limits. Our findings reveal how the IPU's architectural design affects its performance. Moreover, they offer simple mental models to predict an application's performance on the IPU, on the basis of the computation and communication steps it involves. This report is the natural extension to a novel architecture of a continuing effort of ours that focuses on the microbenchmark-based discovery of massively parallel architectures.

91 pages, 21 figures

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• Rethinking the Inception Architecture for Computer Vision
• Aggregated Residual Transformations for Deep Neural Networks

Cited by in corpus (22)

• A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms
• AI and ML Accelerator Survey and Trends
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• Manticore: Hardware-Accelerated RTL Simulation with Static Bulk-Synchronous Parallelism
• Single Flux Quantum Based Ultrahigh Speed Spiking Neuromorphic Processor Architecture
• Effects of round-to-nearest and stochastic rounding in the numerical solution of the heat equation in low precision
• Studying the potential of Graphcore IPUs for applications in Particle Physics
• Parallel Training of Deep Networks with Local Updates
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• Hardware-accelerated Simulation-based Inference of Stochastic Epidemiology Models for COVID-19
• Graphcore C2 Card performance for image-based deep learning application: A Report
• Light-in-the-loop: using a photonics co-processor for scalable training of neural networks
• Improving Neural Network Training in Low Dimensional Random Bases
• Performance of a Geometric Deep Learning Pipeline for HL-LHC Particle Tracking
• Multi-Horizon Forecasting for Limit Order Books: Novel Deep Learning Approaches and Hardware Acceleration using Intelligent Processing Units
• Accelerating Recommender Systems via Hardware "scale-in"
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
• Mini Minds: Exploring Bebeshka and Zlata Baby Models
• ItNet: iterative neural networks with small graphs for accurate, efficient and anytime semantic segmentation
• Comparison of Graphcore IPUs and Nvidia GPUsfor cosmology applications