Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors

Abstract

Although the quest for more accurate solutions is pushing deep learning research towards larger and more complex algorithms, edge devices demand efficient inference and therefore reduction in model size, latency and energy consumption. One technique to limit model size is quantization, which implies using fewer bits to represent weights and biases. Such an approach usually results in a decline in performance. Here, we introduce a method for designing optimally heterogeneously quantized versions of deep neural network models for minimum-energy, high-accuracy, nanosecond inference and fully automated deployment on chip. With a per-layer, per-parameter type automatic quantization procedure, sampling from a wide range of quantizers, model energy consumption and size are minimized while high accuracy is maintained. This is crucial for the event selection procedure in proton-proton collisions at the CERN Large Hadron Collider, where resources are strictly limited and a latency of $\mathcal{O}(1)\mu$s is required. Nanosecond inference and a resource consumption reduced by a factor of 50 when implemented on field-programmable gate array hardware are achieved.

References in corpus (14)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• FINN: A Framework for Fast, Scalable Binarized Neural Network Inference
• FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Review
• Fast inference of deep neural networks in FPGAs for particle physics
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Fast convolutional neural networks on FPGAs with hls4ml
• Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
• JEDI-net: a jet identification algorithm based on interaction networks
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Compressing deep neural networks on FPGAs to binary and ternary precision with HLS4ML
• HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks
• Distance-Weighted Graph Neural Networks on FPGAs for Real-Time Particle Reconstruction in High Energy Physics

Cited by in corpus (43)

• Machine Learning for Microcontroller-Class Hardware: A Review
• Single chip photonic deep neural network with accelerated training
• Fast convolutional neural networks on FPGAs with hls4ml
• Applications and Techniques for Fast Machine Learning in Science
• Autoencoders on FPGAs for real-time, unsupervised new physics detection at 40 MHz at the Large Hadron Collider
• Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation
• Charged particle tracking via edge-classifying interaction networks
• Ps and Qs: Quantization-aware pruning for efficient low latency neural network inference
• DropTrack -- automatic droplet tracking using deep learning for microfluidic applications
• Online-compatible Unsupervised Non-resonant Anomaly Detection
• Graph Neural Networks for Charged Particle Tracking on FPGAs
• PolyLUT: Learning Piecewise Polynomials for Ultra-Low Latency FPGA LUT-based Inference
• Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale
• Nanosecond machine learning event classification with boosted decision trees in FPGA for high energy physics
• LL-GNN: Low Latency Graph Neural Networks on FPGAs for High Energy Physics
• NeuraLUT: Hiding Neural Network Density in Boolean Synthesizable Functions
• Symbolic Regression on FPGAs for Fast Machine Learning Inference
• Fast Muon Tracking with Machine Learning Implemented in FPGA
• Muon trigger with fast Neural Networks on FPGA, a demonstrator
• Nanosecond machine learning regression with deep boosted decision trees in FPGA for high energy physics
• SymbolNet: Neural Symbolic Regression with Adaptive Dynamic Pruning for Compression
• Ultrafast jet classification on FPGAs for the HL-LHC
• Differentiable Earth Mover's Distance for Data Compression at the High-Luminosity LHC
• Noise-induced network topologies
• MetaML: Automating Customizable Cross-Stage Design-Flow for Deep Learning Acceleration
• Real-time Artificial Intelligence for Accelerator Control: A Study at the Fermilab Booster
• Development of a resource-efficient FPGA-based neural network regression model for the ATLAS muon trigger upgrades
• Scaling Up Quantization-Aware Neural Architecture Search for Efficient Deep Learning on the Edge
• da4ml: Distributed Arithmetic for Real-time Neural Networks on FPGAs
• MetaML-Pro: Cross-Stage Design Flow Automation for Efficient Deep Learning Acceleration
• Fast Jet Tagging with MLP-Mixers on FPGAs
• Time-multiplexed In-memory computation scheme for mapping Quantized Neural Networks on hybrid CMOS-OxRAM building blocks
• CGRA4ML: A Hardware/Software Framework to Implement Neural Networks for Scientific Edge Computing
• Investigating Resource-efficient Neutron/Gamma Classification ML Models Targeting eFPGAs
• BitHEP -- The Limits of Low-Precision ML in HEP
• Neural-network-based level-1 trigger upgrade for the SuperCDMS experiment at SNOLAB
• JEDI-linear: Fast and Efficient Graph Neural Networks for Jet Tagging on FPGAs
• It's not a FAD: first results in using Flows for unsupervised Anomaly Detection at 40 MHz at the Large Hadron Collider
• Neural Decoders for Universal Quantum Algorithms
• Real-time graph neural networks on FPGAs for the Belle II electromagnetic calorimeter
• Ultra Fast Calorimeter Simulation with Generative Machine Learning on FPGAs
• Energy-Efficient Real-Time 4-Stage Sleep Classification at 10-Second Resolution
• Distilling particle knowledge for fast reconstruction at high-energy physics experiments