607 citations · 3k across the 51 of their papers we have counts for
8 papers · 1 filter
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
Yixiao Li, Yifan Yu, Qingru Zhang +4
Transformer models have achieved remarkable results in various natural language tasks, but they are often prohibitively large, requiring massive memories and computational resource…
POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models
Korawat Tanwisuth, Shujian Zhang, Huangjie Zheng +2
Through prompting, large-scale pre-trained models have become more expressive and powerful, gaining significant attention in recent years. Though these big models have zero-shot ca…
A Prototype-Oriented Clustering for Domain Shift with Source Privacy
Korawat Tanwisuth, Shujian Zhang, Pengcheng He +1
Unsupervised clustering under domain shift (UCDS) studies how to transfer the knowledge from abundant unlabeled data from multiple source domains to learn the representation of the…
PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance
Qingru Zhang, Simiao Zuo, Chen Liang +4
Large Transformer-based models have exhibited superior performance in various natural language processing and computer vision tasks. However, these models contain enormous amounts…
Diffusion-GAN: Training GANs with Diffusion
Zhendong Wang, Huangjie Zheng, Pengcheng He +2
Generative adversarial networks (GANs) are challenging to train stably, and a promising remedy of injecting instance noise into the discriminator input has not been very effective…
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
Chen Liang, Simiao Zuo, Minshuo Chen +5
The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of ``lottery tickets'', and training a certain collection of them (i.e., a subnetwork) can match t…
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
Simiao Zuo, Chen Liang, Haoming Jiang +5
Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks. Existing works usually for…
On the Variance of the Adaptive Learning Rate and Beyond
Liyuan Liu, Haoming Jiang, Pengcheng He +4
The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization a…