89 citations · 256 across the 28 of their papers we have counts for
28 papers
SLORR: Simple and Efficient In-Training Low-Rank Regularization
David González-Martínez, Shiwei Liu
Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss.…
Learning from the Self-future: On-policy Self-distillation for dLLMs
Yifu Luo, Zeyu Chen, Haoyu Wang +4
On-policy self-distillation (OPSD) has proven effective for post-training large language models (LLMs), yet its application to diffusion LLMs (dLLMs) remains unexplored. Existing O…
(PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetwork
Tianjin Huang, Fang Meng, Li Shen +5
Large-scale neural networks have demonstrated remarkable performance in different domains like vision and language processing, although at the cost of massive computation resources…
REST: Enhancing Group Robustness in DNNs through Reweighted Sparse Training
Jiaxu Zhao, Lu Yin, Shiwei Liu +2
The deep neural network (DNN) has been proven effective in various domains. However, they often struggle to perform well on certain minority groups during inference, despite showin…
Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective
Can Jin, Tianjin Huang, Yihua Zhang +4
The rapid development of large-scale deep learning models questions the affordability of hardware platforms, which necessitates the pruning to reduce their computational and memory…
Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models
Ajay Jaiswal, Shiwei Liu, Tianlong Chen +2
Large pre-trained transformers have been receiving explosive attention in the past few years, due to their wide adaptability for numerous downstream applications via fine-tuning, b…
Enhancing Adversarial Training via Reweighting Optimization Trajectory
Tianjin Huang, Shiwei Liu, Tianlong Chen +6
Despite the fact that adversarial training has become the de facto method for improving the robustness of deep neural networks, it is well-known that vanilla adversarial training s…
Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication
Ajay Jaiswal, Shiwei Liu, Tianlong Chen +2
Graphs are omnipresent and GNNs are a powerful family of neural networks for learning over graphs. Despite their popularity, scaling GNNs either by deepening or widening suffers fr…
The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter
Ajay Jaiswal, Shiwei Liu, Tianlong Chen +1
Large pre-trained transformers are show-stealer in modern-day deep learning, and it becomes crucial to comprehend the parsimonious patterns that exist within them as they grow in s…
Dynamic Sparsity Is Channel-Level Sparsity Learner
Lu Yin, Gen Li, Meng Fang +7
Sparse training has received an upsurging interest in machine learning due to its tantalizing saving potential for the entire training process as well as inference. Dynamic sparse…
Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!
Shiwei Liu, Tianlong Chen, Zhenyu Zhang +4
Sparse Neural Networks (SNNs) have received voluminous attention predominantly due to growing computational and memory footprints of consistently exploding parameter count in large…
Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers
Tianlong Chen, Zhenyu Zhang, Ajay Jaiswal +2
Despite their remarkable achievement, gigantic transformers encounter significant drawbacks, including exorbitant computational and memory footprints during training, as well as se…
Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers
Shiwei Liu, Zhangyang Wang
This article does not propose any novel algorithm or new hardware for sparsity. Instead, it aims to serve the "common good" for the increasingly prosperous Sparse Neural Network (S…
Dynamic Sparse Network for Time Series Classification: Learning What to "see''
Qiao Xiao, Boqian Wu, Yu Zhang +4
The receptive field (RF), which determines the region of time series to be ``seen'' and used, is critical to improve the performance for time series classification (TSC). However,…
Lottery Pools: Winning More by Interpolating Tickets without Increasing Training or Inference Cost
Lu Yin, Shiwei Liu, Meng Fang +3
Lottery tickets (LTs) is able to discover accurate and sparse subnetworks that could be trained in isolation to match the performance of dense networks. Ensemble, in parallel, is o…
More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity
Shiwei Liu, Tianlong Chen, Xiaohan Chen +7
Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems to…
Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training
Lu Yin, Vlado Menkovski, Meng Fang +5
Recent works on sparse neural network training (sparse training) have shown that a compelling trade-off between performance and efficiency can be achieved by training intrinsically…
Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance
Shiwei Liu, Yuesong Tian, Tianlong Chen +1
Generative adversarial networks (GANs) have received an upsurging interest since being proposed due to the high quality of the generated data. While achieving increasingly impressi…
The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training
Shiwei Liu, Tianlong Chen, Xiaohan Chen +4
Random pruning is arguably the most naive way to attain sparsity in neural networks, but has been deemed uncompetitive by either post-training pruning or sparse training. In this p…
Achieving Personalized Federated Learning with Sparse Local Models
Tiansheng Huang, Shiwei Liu, Li Shen +3
Federated learning (FL) is vulnerable to heterogeneously distributed data, since a common global model in FL may not adapt to the heterogeneous data distribution of each user. To c…
Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
Shiwei Liu, Tianlong Chen, Zahra Atashgahi +6
The success of deep ensembles on improving predictive performance, uncertainty estimation, and out-of-distribution robustness has been extensively studied in the machine learning l…
Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
Shiwei Liu, Tianlong Chen, Xiaohan Chen +7
Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and…
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu +1
In this paper, we introduce a new perspective on training deep neural networks capable of state-of-the-art performance without the need for the expensive over-parameterization by p…
Selfish Sparse RNN Training
Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei +1
Sparse neural networks have been widely applied to reduce the computational demands of training and deploying over-parameterized deep neural networks. For inference acceleration, m…