FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference

Abstract

Deep learning models typically use single-precision (FP32) floating point data types for representing activations and weights, but a slew of recent research work has shown that computations with reduced-precision data types (FP16, 16-bit integers, 8-bit integers or even 4- or 2-bit integers) are enough to achieve same accuracy as FP32 and are much more efficient. Therefore, we designed fbgemm, a high-performance kernel library, from ground up to perform high-performance quantized inference on current generation CPUs. fbgemm achieves efficiency by fusing common quantization operations with a high-performance gemm implementation and by shape- and size-specific kernel code generation at runtime. The library has been deployed at Facebook, where it delivers greater than 2x performance gains with respect to our current production baseline.

Cited by in corpus (8)

• FEATHER: A Reconfigurable Accelerator with Data Reordering Support for Low-Cost On-Chip Dataflow Switching
• FPUS23: An Ultrasound Fetus Phantom Dataset with Deep Neural Network Evaluations for Fetus Orientations, Fetal Planes, and Anatomical Features
• Model Pruning Based on Quantified Similarity of Feature Maps
• Revisiting Neural Retrieval on Accelerators
• Efficient Soft-Error Detection for Low-precision Deep Learning Recommendation Models
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale
• The NiuTrans System for the WMT21 Efficiency Task
• Supporting Massive DLRM Inference Through Software Defined Memory