Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inference

Abstract

Real-time Deep Neural Network (DNN) inference with low-latency requirement has become increasingly important for numerous applications in both cloud computing (e.g., Apple's Siri) and edge computing (e.g., Google/Waymo's driverless car). FPGA-based DNN accelerators have demonstrated both superior flexibility and performance; in addition, for real-time inference with low batch size, FPGA is expected to achieve further performance improvement. However, the performance gain from the single-FPGA design is obstructed by the limited on-chip resource. In this paper, we employ multiple FPGAs to cooperatively run DNNs with the objective of achieving super-linear speed-up against single-FPGA design. In implementing such systems, we found two barriers that hinder us from achieving the design goal: (1) the lack of a clear partition scheme for each DNN layer to fully exploit parallelism, and (2) the insufficient bandwidth between the off-chip memory and the accelerator due to the growing size of DNNs. To tackle these issues, we propose a general framework, "Super-LIP", which can support different kinds of DNNs. In this paper, we take Convolutional Neural Network (CNN) as a vehicle to illustrate Super-LIP. We first formulate an accurate system-level model to support the exploration of best partition schemes. Then, we develop a novel design methodology to effectively alleviate the heavy loads on memory bandwidth by moving traffic from memory bus to inter-FPGA links. We implement Super-LIP based on ZCU102 FPGA boards. Results demonstrate that Super-LIP with 2 FPGAs can achieve 3.48x speedup, compared to the state-of-the-art single-FPGA design. What is more, as the number of FPGAs scales up, the system latency can be further reduced while maintaining high energy efficiency.

arXiv commentAccepted by ESWEEK CODES+ISSS 2019, nominated as best paper. Appear in ACM TECS

References in corpus (5)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Recent Trends in Deep Learning Based Natural Language Processing
• On the Universal Approximability and Complexity Bounds of Quantized ReLU Neural Networks
• Hardware/Software Co-Exploration of Neural Architectures
• Accuracy vs. Efficiency: Achieving Both through FPGA-Implementation Aware Neural Architecture Search

Cited by in corpus (15)

• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• When Machine Learning Meets Quantum Computers: A Case Study
• A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference
• Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks
• ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learning
• Standing on the Shoulders of Giants: Hardware and Neural Architecture Co-Search with Hot Start
• Brain-inspired Cognition in Next Generation Racetrack Memories
• Exploration of Quantum Neural Architecture by Mixing Quantum Neuron Designs
• Can Noise on Qubits Be Learned in Quantum Neural Network? A Case Study on QuantumFlow
• Dancing along Battery: Enabling Transformer with Run-time Reconfigurability on Mobile Devices
• Device-Circuit-Architecture Co-Exploration for Computing-in-Memory Neural Accelerators
• When Single Event Upset Meets Deep Neural Networks: Observations, Explorations, and Remedies
• PIRM: Processing In Racetrack Memories
• Enabling OpenMP Task Parallelism on Multi-FPGAs
• Low Overhead Online Data Flow Tracking for Intermittently Powered Non-volatile FPGAs