27 citations · 113 across the 27 of their papers we have counts for
8 papers · 1 filter
Efficient Long Sequence Modeling via State Space Augmented Transformer
Simiao Zuo, Xiaodong Liu, Jian Jiao +4
Transformer models have achieved superior performance in various natural language processing tasks. However, the quadratic computational cost of the attention mechanism limits its…
Less is More: Task-aware Layer-wise Distillation for Language Model Compression
Chen Liang, Simiao Zuo, Qingru Zhang +3
Layer-wise distillation is a powerful tool to compress large models (i.e. teacher models) into small ones (i.e., student models). The student distills knowledge from the teacher by…
Context-Aware Query Rewriting for Improving Users' Search Experience on E-commerce Websites
Simiao Zuo, Qingyu Yin, Haoming Jiang +4
E-commerce queries are often short and ambiguous. Consequently, query understanding often uses query rewriting to disambiguate user-input queries. While using e-commerce search too…
DiP-GNN: Discriminative Pre-Training of Graph Neural Networks
Simiao Zuo, Haoming Jiang, Qingyu Yin +3
Graph neural network (GNN) pre-training methods have been proposed to enhance the power of GNNs. Specifically, a GNN is first pre-trained on a large-scale unlabeled graph and then…
Differentially Private Estimation of Hawkes Process
Simiao Zuo, Tianyi Liu, Tuo Zhao +1
Point process models are of great importance in real world applications. In certain critical applications, estimation of point process models involves large amounts of sensitive pe…
PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance
Qingru Zhang, Simiao Zuo, Chen Liang +4
Large Transformer-based models have exhibited superior performance in various natural language processing and computer vision tasks. However, these models contain enormous amounts…
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation
Simiao Zuo, Qingru Zhang, Chen Liang +3
Pre-trained language models have demonstrated superior performance in various natural language processing tasks. However, these models usually contain hundreds of millions of param…
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
Chen Liang, Haoming Jiang, Simiao Zuo +5
Recent research has shown the existence of significant redundancy in large Transformer models. One can prune the redundant parameters without significantly sacrificing the generali…