109 citations · 178 across the 14 of their papers we have counts for
4 papers · 1 filter
A Length-Extrapolatable Transformer
Yutao Sun, Li Dong, Barun Patra +6
Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define…
Structured Prompting: Scaling In-Context Learning to 1,000 Examples
Yaru Hao, Yutao Sun, Li Dong +3
Large language models have exhibited intriguing in-context learning capability, achieving promising zero- and few-shot performance without updating the parameters. However, convent…
Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
Damai Dai, Yutao Sun, Li Dong +4
Large pretrained language models have shown surprising in-context learning (ICL) ability. With a few demonstration input-label pairs, they can predict the label for an unseen input…
Prototypical Calibration for Few-shot Learning of Language Models
Zhixiong Han, Yaru Hao, Li Dong +2
In-context learning of GPT-like models has been recognized as fragile across different hand-crafted templates, and demonstration permutations. In this work, we propose prototypical…