563 citations · 813 across the 25 of their papers we have counts for
9 papers · 1 filter
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BigScience Workshop, :, Teven Le Scao +391
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to wi…
InforMask: Unsupervised Informative Masking for Language Model Pretraining
Nafis Sadeq, Canwen Xu, Julian McAuley
Masked language modeling is widely used for pretraining large language models for natural language understanding (NLU). However, random masking is suboptimal, allocating an equal m…
Efficiently Tuned Parameters are Task Embeddings
Wangchunshu Zhou, Canwen Xu, Julian McAuley
Intermediate-task transfer can benefit a wide range of NLP tasks with properly selected source datasets. However, it is computationally infeasible to experiment with all intermedia…
LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval
Canwen Xu, Daya Guo, Nan Duan +1
In this paper, we propose LaPraDoR, a pretrained dual-tower dense retriever that does not require any supervised data for training. Specifically, we first present Iterative Contras…
Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot Classification
Han Wang, Canwen Xu, Julian McAuley
Prompt-based learning (i.e., prompting) is an emerging paradigm for exploiting knowledge learned by a pretrained language model. In this paper, we propose Automatic Multi-Label Pro…
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Stephen H. Bach, Victor Sanh, Zheng-Xin Yong +24
PromptSource is a system for creating, sharing, and using natural language prompts. Prompts are functions that map an example from a dataset to a natural language input and target…
Leashing the Inner Demons: Self-Detoxification for Language Models
Canwen Xu, Zexue He, Zhankui He +1
Language models (LMs) can reproduce (or amplify) toxic language seen during training, which poses a risk to their practical application. In this paper, we conduct extensive experim…
A Survey on Model Compression and Acceleration for Pretrained Language Models
Canwen Xu, Julian McAuley
Despite achieving state-of-the-art performance on many NLP tasks, the high energy cost and long inference delay prevent Transformer-based pretrained language models (PLMs) from see…
A Survey on Dynamic Neural Networks for Natural Language Processing
Canwen Xu, Julian McAuley
Effectively scaling large Transformer models is a main driver of recent advances in natural language processing. Dynamic neural networks, as an emerging research direction, are cap…