544 citations · 552 across the 8 of their papers we have counts for
3 papers · 1 filter
Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU
Jafar Badour, Maurice van Keulen, Elena Mocanu
Deep neural networks continue to grow in parameter count, driving up training and inference cost on GPUs. Sparse neural networks and Dynamic Sparse Training (DST) promise to reduce…
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
Boqian Wu, Qiao Xiao, Patrik Okanovic +6
Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained r…
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
Qiao Xiao, Boqian Wu, Patrik Okanovic +6
Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model…