205 citations · 276 across the 7 of their papers we have counts for
5 papers · 1 filter
Is Finer Better? The Limits of Microscaling Formats in Large Language Models
Andrea Fasoli, Monodeep Kar, Chi-Chun Liu +4
Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient train…
ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training
Chia-Yu Chen, Jiamin Ni, Songtao Lu +8
Large-scale distributed training of Deep Neural Networks (DNNs) on state-of-the-art platforms is expected to be severely communication constrained. To overcome this limitation, num…
A Comprehensive Survey on Hardware-Aware Neural Architecture Search
Hadjer Benmeziane, Kaoutar El Maghraoui, Hamza Ouarnoughi +3
Neural Architecture Search (NAS) methods have been growing in popularity. These techniques have been fundamental to automate and speed up the time consuming and error-prone process…
Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks
Charbel Sakr, Naigang Wang, Chia-Yu Chen +4
Efforts to reduce the numerical precision of computations in deep learning training have yielded systems that aggressively quantize weights and activations, yet employ wide high-pr…
Training Deep Neural Networks with 8-bit Floating Point Numbers
Naigang Wang, Jungwook Choi, Daniel Brand +2
The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision --…