941 citations · 5.4k across the 87 of their papers we have counts for
28 papers · 1 filter
A Length-Extrapolatable Transformer
Yutao Sun, Li Dong, Barun Patra +6
Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define…
Structured Prompting: Scaling In-Context Learning to 1,000 Examples
Yaru Hao, Yutao Sun, Li Dong +3
Large language models have exhibited intriguing in-context learning capability, achieving promising zero- and few-shot performance without updating the parameters. However, convent…
Language Models as Inductive Reasoners
Zonglin Yang, Li Dong, Xinya Du +5
Inductive reasoning is a core component of human intelligence. In the past research of inductive reasoning within computer science, formal language is used as representations of kn…
Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
Damai Dai, Yutao Sun, Li Dong +4
Large pretrained language models have shown surprising in-context learning (ICL) ability. With a few demonstration input-label pairs, they can predict the label for an unseen input…
GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator
Jian Yang, Shuming Ma, Li Dong +7
Pre-trained models have achieved remarkable success in natural language processing (NLP). However, existing pre-training methods underutilize the benefits of language understanding…
Optimizing Prompts for Text-to-Image Generation
Yaru Hao, Zewen Chi, Li Dong +1
Well-designed prompts can guide text-to-image models to generate amazing images. However, the performant prompts are often model-specific and misaligned with user input. Instead of…