2.5k citations · 4.3k across the 50 of their papers we have counts for
51 papers
GRIN: GRadient-INformed MoE
Liyuan Liu, Young Jin Kim, Shuohang Wang +14
Mixture-of-Experts (MoE) models scale more effectively than dense models due to sparse computation through expert routing, selectively activating only a small subset of expert modu…
GRILL: Grounded Vision-language Pre-training via Aligning Text and Image Regions
Woojeong Jin, Subhabrata Mukherjee, Yu Cheng +5
Generalization to unseen tasks is an important ability for few-shot learners to achieve better zero-/few-shot performance on diverse tasks. However, such generalization to vision-l…
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
Zhihong Shao, Yeyun Gong, Yelong Shen +3
Large language models are powerful text processors and reasoners, but are still subject to limitations including outdated knowledge and hallucinations, which necessitates connectin…
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Zhibin Gou, Zhihong Shao, Yeyun Gong +4
Recent developments in large language models (LLMs) have been impressive. However, these models sometimes show inconsistencies and problematic behavior, such as hallucinating facts…
AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Tong Wu, Zhihao Fan, Xiao Liu +9
Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generatio…
In-Context Learning Unlocked for Diffusion Models
Zhendong Wang, Yifan Jiang, Yadong Lu +5
We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models. Given a pair of task-specific example images, such as depth from/to…
Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models
Jiashuo Sun, Yi Luo, Yeyun Gong +4
Large language models (LLMs) can achieve highly effective performance on various reasoning tasks by incorporating step-by-step chain-of-thought (CoT) prompting as demonstrations. H…
What Matters In The Structured Pruning of Generative Language Models?
Michael Santacroce, Zixin Wen, Yelong Shen +1
Auto-regressive large language models such as GPT-3 require enormous computational resources to use. Traditionally, structured pruning methods are employed to reduce resource usage…
Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models
Zhihong Shao, Yeyun Gong, Yelong Shen +3
Large language models can perform various reasoning tasks by using chain-of-thought prompting, which guides them to find answers through step-by-step demonstrations. However, the q…
Generation-Augmented Query Expansion For Code Retrieval
Dong Li, Yelong Shen, Ruoming Jin +3
Pre-trained language models have achieved promising success in code retrieval tasks, where a natural language documentation query is given to find the most relevant existing code s…
Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise
Zhenghao Lin, Yeyun Gong, Yelong Shen +5
In this paper, we introduce a novel dIffusion language modEl pre-training framework for text generation, which we call GENIE. GENIE is a large-scale pretrained diffusion language m…
GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation
Biyang Guo, Yeyun Gong, Yelong Shen +4
We introduce GENIUS: a conditional text generation model using sketches as input, which can fill in the missing contexts for a given sketch (key information consisting of textual s…
Soft-Labeled Contrastive Pre-training for Function-level Code Representation
Xiaonan Li, Daya Guo, Yeyun Gong +6
Code contrastive pre-training has recently achieved significant progress on code-related tasks. In this paper, we present \textbf{SCodeR}, a \textbf{S}oft-labeled contrastive pre-t…
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Kun Zhou, Yeyun Gong, Xiao Liu +8
Sampling proper negatives from a large document pool is vital to effectively train a dense retrieval model. However, existing negative sampling strategies suffer from the uninforma…
Explanations from Large Language Models Make Small Reasoners Better
Shiyang Li, Jianshu Chen, Yelong Shen +9
Integrating free-text explanations to in-context learning of large language models (LLM) is shown to elicit strong reasoning capabilities along with reasonable explanations. In thi…
Joint Generator-Ranker Learning for Natural Language Generation
Weizhou Shen, Yeyun Gong, Yelong Shen +4
Generate-then-rank is a widely used mechanism for text generation, where a generator produces multiple text candidates and a ranker chooses the best one among the text candidates.…
A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation
Weizhen Qi, Yeyun Gong, Yelong Shen +6
Non-Autoregressive generation is a sequence generation paradigm, which removes the dependency between target tokens. It could efficiently reduce the text generation latency with pa…
CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing
Chen Liang, Pengcheng He, Yelong Shen +2
Model ensemble is a popular approach to produce a low-variance and well-generalized model. However, it induces large memory and inference costs, which are often not affordable for…
Controllable Natural Language Generation with Contrastive Prefixes
Jing Qian, Li Dong, Yelong Shen +2
To guide the generation of large pretrained language models (LM), previous work has focused on directly fine-tuning the language model or utilizing an attribute discriminator. In t…
CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search
Xiaonan Li, Yeyun Gong, Yelong Shen +7
In this paper, we propose the CodeRetriever model, which learns the function-level code semantic representations through large-scale code-text contrastive pre-training. We adopt tw…
Knowledge-Grounded Dialogue Generation with a Unified Knowledge Representation
Yu Li, Baolin Peng, Yelong Shen +4
Knowledge-grounded dialogue systems are challenging to build due to the lack of training data and heterogeneous knowledge sources. Existing systems perform poorly on unseen topics…
A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models
Woojeong Jin, Yu Cheng, Yelong Shen +2
Large pre-trained vision-language (VL) models can learn a new task with a handful of examples and generalize to a new task without fine-tuning. However, these VL models are hard to…
Adversarial Retriever-Ranker for dense text retrieval
Hang Zhang, Yeyun Gong, Yelong Shen +3
Current dense text retrieval models face two typical challenges. First, they adopt a siamese dual-encoder architecture to encode queries and documents independently for fast indexi…
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis +5
An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larg…