8 citations · 11 across the 4 of their papers we have counts for
7 papers
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation
Simiao Zuo, Qingru Zhang, Chen Liang +3
Pre-trained language models have demonstrated superior performance in various natural language processing tasks. However, these models usually contain hundreds of millions of param…
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
Chen Liang, Haoming Jiang, Simiao Zuo +5
Recent research has shown the existence of significant redundancy in large Transformer models. One can prune the redundant parameters without significantly sacrificing the generali…
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
Chen Liang, Simiao Zuo, Minshuo Chen +5
The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of ``lottery tickets'', and training a certain collection of them (i.e., a subnetwork) can match t…
A Hypergradient Approach to Robust Regression without Correspondence
Yujia Xie, Yixiu Mao, Simiao Zuo +4
We consider a variant of regression problem, where the correspondence between input and output data is not available. Such shuffled data is commonly observed in many real world pro…
Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach
Yue Yu, Simiao Zuo, Haoming Jiang +3
Fine-tuned pre-trained language models (LMs) have achieved enormous success in many natural language processing (NLP) tasks, but they still require excessive labeled data in the fi…
Transformer Hawkes Process
Simiao Zuo, Haoming Jiang, Zichong Li +2
Modern data acquisition routinely produce massive amounts of event sequence data in various domains, such as social media, healthcare, and financial markets. These data often exhib…