257 citations · 621 across the 31 of their papers we have counts for
7 papers · 1 filter
Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
Damai Dai, Yutao Sun, Li Dong +4
Large pretrained language models have shown surprising in-context learning (ICL) ability. With a few demonstration input-label pairs, they can predict the label for an unseen input…
Calibrating Factual Knowledge in Pretrained Language Models
Qingxiu Dong, Damai Dai, Yifan Song +3
Previous literature has proved that Pretrained Language Models (PLMs) can store factual knowledge. However, we find that facts stored in the PLMs are not always correct. It motivat…
Neural Knowledge Bank for Pretrained Transformers
Damai Dai, Wenbin Jiang, Qingxiu Dong +3
The ability of pretrained Transformers to remember factual knowledge is essential but still limited for existing models. Inspired by existing work that regards Feed-Forward Network…
Robust Fine-tuning via Perturbation and Interpolation from In-batch Instances
Shoujie Tong, Qingxiu Dong, Damai Dai +4
Fine-tuning pretrained language models (PLMs) on downstream tasks has become common practice in natural language processing. However, most of the PLMs are vulnerable, e.g., they ar…
StableMoE: Stable Routing Strategy for Mixture of Experts
Damai Dai, Li Dong, Shuming Ma +4
The Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead. We point out that existing learning-to-route MoE metho…
Mixture of Experts for Biomedical Question Answering
Damai Dai, Wenbin Jiang, Jiyuan Zhang +5
Biomedical Question Answering (BQA) has attracted increasing attention in recent years due to its promising application prospect. It is a challenging task because the biomedical qu…
On the Representation Collapse of Sparse Mixture of Experts
Zewen Chi, Li Dong, Shaohan Huang +9
Sparse mixture of experts provides larger model capacity while requiring a constant computational overhead. It employs the routing mechanism to distribute input tokens to the best-…