1.8k citations · 3.8k across the 47 of their papers we have counts for
9 papers · 1 filter
NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework
Xingcheng Yao, Yanan Zheng, Xiaocong Yang +1
Pretrained language models have become the standard approach for many NLP tasks due to strong performance, but they are very expensive to train. We propose a simple and efficient l…
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
Xiao Liu, Kaixuan Ji, Yicheng Fu +4
Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU…
FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding
Yanan Zheng, Jing Zhou, Yujie Qian +7
The few-shot natural language understanding (NLU) task has attracted much recent attention. However, prior methods have been evaluated under a disparate set of protocols, which hin…
FlipDA: Effective and Robust Data Augmentation for Few-Shot Learning
Jing Zhou, Yanan Zheng, Jie Tang +2
Most previous methods for text data augmentation are limited to simple tasks and weak baselines. We explore data augmentation on hard tasks (i.e., few-shot natural language underst…
Distribution Matching for Rationalization
Yongfeng Huang, Yujun Chen, Yulun Du +1
The task of rationalization aims to extract pieces of input text as rationales to justify neural network predictions on text classification tasks. By definition, rationales represe…
FastMoE: A Fast Mixture-of-Expert Training System
Jiaao He, Jiezhong Qiu, Aohan Zeng +3
Mixture-of-Expert (MoE) presents a strong potential in enlarging the size of language model to trillions of parameters. However, training trillion-scale MoE requires algorithm and…
Controllable Generation from Pre-trained Language Models via Inverse Prompting
Xu Zou, Da Yin, Qingyang Zhong +4
Large-scale pre-trained language models have demonstrated strong capabilities of generating realistic text. However, it remains challenging to control the generation results. Previ…
GPT Understands, Too
Xiao Liu, Yanan Zheng, Zhengxiao Du +4
Prompting a pretrained language model with natural language patterns has been proved effective for natural language understanding (NLU). However, our preliminary study reveals that…
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Zhengxiao Du, Yujie Qian, Xiao Liu +4
There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). How…