27 citations · 113 across the 27 of their papers we have counts for
12 papers · 1 filter
m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encoder
Yaoxiang Wang, Simiao Zuo, Qingguo Hu +4
Embedding models are pivotal in industrial information retrieval systems like search and advertising. However, existing pretrained models often exhibit fixed architectures and embe…
Task Oriented In-Domain Data Augmentation
Xiao Liang, Xinyu Hu, Simiao Zuo +5
Large Language Models (LLMs) have shown superior performance in various applications and fields. To achieve better performance on specialized domains such as law and advertisement,…
Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning
Yanda Chen, Chandan Singh, Xiaodong Liu +4
Large language models (LLMs) often generate convincing, fluent explanations. However, different from humans, they often generate inconsistent explanations on different inputs. For…
Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editing
Xinyu Hu, Pengfei Tang, Simiao Zuo +5
Large language models (LLMs) have made impressive progress in natural language processing. These models rely on proper human instructions (or prompts) to generate suitable response…
Efficient Long Sequence Modeling via State Space Augmented Transformer
Simiao Zuo, Xiaodong Liu, Jian Jiao +4
Transformer models have achieved superior performance in various natural language processing tasks. However, the quadratic computational cost of the attention mechanism limits its…
Less is More: Task-aware Layer-wise Distillation for Language Model Compression
Chen Liang, Simiao Zuo, Qingru Zhang +3
Layer-wise distillation is a powerful tool to compress large models (i.e. teacher models) into small ones (i.e., student models). The student distills knowledge from the teacher by…
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation
Simiao Zuo, Qingru Zhang, Chen Liang +3
Pre-trained language models have demonstrated superior performance in various natural language processing tasks. However, these models usually contain hundreds of millions of param…
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
Chen Liang, Haoming Jiang, Simiao Zuo +5
Recent research has shown the existence of significant redundancy in large Transformer models. One can prune the redundant parameters without significantly sacrificing the generali…
Taming Sparsely Activated Transformer with Stochastic Experts
Simiao Zuo, Xiaodong Liu, Jian Jiao +5
Sparsely activated models (SAMs), such as Mixture-of-Experts (MoE), can easily scale to have outrageously large amounts of parameters without significant increase in computational…
Self-Training with Differentiable Teacher
Simiao Zuo, Yue Yu, Chen Liang +5
Self-training achieves enormous success in various semi-supervised and weakly-supervised learning tasks. The method can be interpreted as a teacher-student framework, where the tea…
ARCH: Efficient Adversarial Regularized Training with Caching
Simiao Zuo, Chen Liang, Haoming Jiang +5
Adversarial regularization can improve model generalization in many natural language processing tasks. However, conventional approaches are computationally expensive since they nee…
Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach
Yue Yu, Simiao Zuo, Haoming Jiang +3
Fine-tuned pre-trained language models (LMs) have achieved enormous success in many natural language processing (NLP) tasks, but they still require excessive labeled data in the fi…