1.2k citations · 1.4k across the 8 of their papers we have counts for
8 papers
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
Sheng Shen, Le Hou, Yanqi Zhou +17
Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost.…
Large Language Models Can Self-Improve
Jiaxin Huang, Shixiang Shane Gu, Le Hou +4
Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improv…
Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation
Ziqi Wang, Yuexin Wu, Frederick Liu +5
Knowledge distillation is one of the primary methods of transferring knowledge from large to small models. However, it requires massive task-specific data, which may not be plausib…
Scaling Instruction-Finetuned Language Models
Hyung Won Chung, Le Hou, Shayne Longpre +32
Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we expl…
EncT5: A Framework for Fine-tuning T5 as Non-autoregressive Models
Frederick Liu, Terry Huang, Shihang Lyu +3
Pre-trained encoder-decoder transformer architectures have become increasingly popular recently with the advent of T5 models. T5 has also become more favorable over other architect…
On the Transformer Growth for Progressive BERT Training
Xiaotao Gu, Liyuan Liu, Hongkun Yu +3
Due to the excessive cost of large-scale language model pre-training, considerable efforts have been made to train BERT progressively -- start from an inferior but low-cost model a…
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Zhiqing Sun, Hongkun Yu, Xiaodan Song +3
Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters. However, these models suffer from he…
Generating Representative Headlines for News Stories
Xiaotao Gu, Yuning Mao, Jiawei Han +7
Millions of news articles are published online every day, which can be overwhelming for readers to follow. Grouping articles that are reporting the same event into news stories is…