233 citations · 266 across the 5 of their papers we have counts for
3 papers · 1 filter
Accelerating Transformer Inference and Training with 2:4 Activation Sparsity
Daniel Haziza, Timothy Chou, Dhruv Choudhary +7
In this paper, we demonstrate how to leverage 2:4 sparsity, a popular hardware-accelerated GPU sparsity pattern, to activations to accelerate large language model training and infe…
Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training
Jared Fernandez, Luca Wehrstedt, Leonid Shamis +5
Dramatic increases in the capabilities of neural network models in recent years are driven by scaling model size, training data, and corresponding computational resources. To devel…
PyTorch-BigGraph: A Large-scale Graph Embedding System
Adam Lerer, Ledell Wu, Jiajun Shen +4
Graph embedding methods produce unsupervised node features from graphs that can then be used for a variety of machine learning tasks. Modern graphs, particularly in industrial appl…