How to keep pushing ML accelerator performance? Know your rooflines!

Abstract

The rapidly growing importance of Machine Learning (ML) applications, coupled with their ever-increasing model size and inference energy footprint, has created a strong need for specialized ML hardware architectures. Numerous ML accelerators have been explored and implemented, primarily to increase task-level throughput per unit area and reduce task-level energy consumption. This paper surveys key trends toward these objectives for more efficient ML accelerators and provides a unifying framework to understand how compute and memory technologies/architectures interact to enhance system-level efficiency and performance. To achieve this, the paper introduces an enhanced version of the roofline model and applies it to ML accelerators as an effective tool for understanding where various execution regimes fall within roofline bounds and how to maximize performance and efficiency under the rooline. Key concepts are illustrated with examples from state-of-the-art designs, with a view towards open research opportunities to further advance accelerator performance.

in IEEE Journal of Solid-State Circuits, 2025

References in corpus (10)

• NVIDIA Tensor Core Programmability, Performance & Precision
• A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference
• A Microprocessor implemented in 65nm CMOS with Configurable and Bit-scalable Accelerator for Programmable In-memory Computing
• Interstellar: Using Halide's Scheduling Language to Analyze DNN Accelerators
• NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing
• Mind Mappings: Enabling Efficient Algorithm-Accelerator Mapping Space Search
• Marsellus: A Heterogeneous RISC-V AI-IoT End-Node SoC with 2-to-8b DNN Acceleration and 30%-Boost Adaptive Body Biasing
• Occamy: A 432-Core Dual-Chiplet Dual-HBM2E 768-DP-GFLOP/s RISC-V System for 8-to-64-bit Dense and Sparse Computing in 12nm FinFET
• Stream: Design Space Exploration of Layer-Fused DNNs on Heterogeneous Dataflow Accelerators
• Microscaling Data Formats for Deep Learning