95 citations · 198 across the 9 of their papers we have counts for
9 papers
HCCL: Collective Communication for Meta Training and Inference Accelerators
Wesley Bland, Tiago Antunes, Lars Paul Huse +63
We present HCCL, a collective communication library co-designed with Meta's MTIA 300 accelerator, the first Meta chip to integrate backend networking directly on chip package. MTIA…
ST-MoE: Designing Stable and Transferable Sparse Expert Models
Barret Zoph, Irwan Bello, Sameer Kumar +5
Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy e…
Highly Available Data Parallel ML training on Mesh Networks
Sameer Kumar, Norm Jouppi
Data parallel ML models can take several days or weeks to train on several accelerators. The long duration of training relies on the cluster of resources to be available for the jo…
Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour
Arissa Wongpanich, Hieu Pham, James Demmel +4
EfficientNets are a family of state-of-the-art image classification models based on efficiently scaled convolutional neural networks. Currently, EfficientNets can take on the order…
Exploring the limits of Concurrency in ML Training on Google TPUs
Sameer Kumar, James Bradbury, Cliff Young +16
Recent results in language understanding using neural networks have required training hardware of unprecedentedscale, with thousands of chips cooperating on a single training run.…
Scale MLPerf-0.6 models on Google TPU-v3 Pods
Sameer Kumar, Victor Bitorff, Dehao Chen +9
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6 training benchmark demonstrates the scalability of a suite of industry relevant ML models. MLPerf defin…