27 citations · 113 across the 27 of their papers we have counts for
6 papers · 1 filter
Taming Sparsely Activated Transformer with Stochastic Experts
Simiao Zuo, Xiaodong Liu, Jian Jiao +5
Sparsely activated models (SAMs), such as Mixture-of-Experts (MoE), can easily scale to have outrageously large amounts of parameters without significant increase in computational…
Adversarially Regularized Policy Learning Guided by Trajectory Optimization
Zhigen Zhao, Simiao Zuo, Tuo Zhao +1
Recent advancement in combining trajectory optimization with function approximation (especially neural networks) shows promise in learning complex control policies for diverse task…
Self-Training with Differentiable Teacher
Simiao Zuo, Yue Yu, Chen Liang +5
Self-training achieves enormous success in various semi-supervised and weakly-supervised learning tasks. The method can be interpreted as a teacher-student framework, where the tea…
ARCH: Efficient Adversarial Regularized Training with Caching
Simiao Zuo, Chen Liang, Haoming Jiang +5
Adversarial regularization can improve model generalization in many natural language processing tasks. However, conventional approaches are computationally expensive since they nee…
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
Chen Liang, Simiao Zuo, Minshuo Chen +5
The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of ``lottery tickets'', and training a certain collection of them (i.e., a subnetwork) can match t…
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
Simiao Zuo, Chen Liang, Haoming Jiang +5
Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks. Existing works usually for…