1 citations · 1 across the 12 of their papers we have counts for
6 papers · 1 filter
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
Boqian Wu, Qiao Xiao, Patrik Okanovic +6
Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained r…
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
Qiao Xiao, Boqian Wu, Patrik Okanovic +6
Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model…
Batch Matrix-form Equations and Implementation of Multilayer Perceptrons
Wieger Wesselink, Bram Grooten, Huub van de Wetering +2
Multilayer perceptrons (MLPs) remain fundamental to modern deep learning, yet their algorithmic details are rarely presented in complete, explicit \emph{batch matrix-form}. Rather,…
Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsity
Qiao Xiao, Boqian Wu, Andrey Poddubnyy +4
Federated learning (FL) enables collaborative model training across decentralized clients while preserving data privacy, leveraging aggregated updates to build robust global models…
NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling
Bram Grooten, Farid Hasanov, Chenxiang Zhang +9
Model ensembles have long been a cornerstone for improving generalization and robustness in deep learning. However, their effectiveness often comes at the cost of substantial compu…
Nerva: a Truly Sparse Implementation of Neural Networks
Wieger Wesselink, Bram Grooten, Qiao Xiao +2
We introduce Nerva, a fast neural network library under development in C++. It supports sparsity by using the sparse matrix operations of Intel's Math Kernel Library (MKL), which e…