439 citations · 458 across the 2 of their papers we have counts for
4 papers
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
Trevor Gale, Deepak Narayanan, Cliff Young +1
We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE) training on GPUs. Our system is motivated by the limitations of current frameworks, which restrict the dynami…
Sparse GPU Kernels for Deep Learning
Trevor Gale, Matei Zaharia, Cliff Young +1
Scientific workloads have traditionally exploited high levels of sparsity to accelerate computation and reduce memory requirements. While deep neural networks can be made sparse, a…
Fast Sparse ConvNets
Erich Elsen, Marat Dukhan, Trevor Gale +1
Historically, the pursuit of efficient inference has been one of the driving forces behind research into new deep learning architectures and building blocks. Some recent examples i…
The State of Sparsity in Deep Neural Networks
Trevor Gale, Erich Elsen, Sara Hooker
We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to…