607 citations · 3k across the 51 of their papers we have counts for
51 papers
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
Yixiao Li, Yifan Yu, Chen Liang +4
Quantization is an indispensable technique for serving Large Language Models (LLMs) and has recently found its way into LoRA fine-tuning. In this work we focus on the scenario wher…
Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective
Ming Zhong, Chenxin An, Weizhu Chen +2
Large Language Models (LLMs) inherently encode a wealth of knowledge within their parameters through pre-training on extensive corpora. While prior research has delved into operati…
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
Yung-Sung Chuang, Yujia Xie, Hongyin Luo +3
Despite their impressive capabilities, large language models (LLMs) are prone to hallucinations, i.e., generating content that deviates from facts seen during pretraining. We propo…
Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
Zekun Li, Baolin Peng, Pengcheng He +1
Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following, becoming increasingly crucial across various applications. However, this capability…
Summaries, Highlights, and Action items: Design, implementation and evaluation of an LLM-powered meeting recap system
Sumit Asthana, Sagih Hilleli, Pengcheng He +1
Meetings play a critical infrastructural role in coordinating work. The recent surge of hybrid and remote meetings in computer-mediated spaces has led to new problems (e.g., more t…
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
Yixiao Li, Yifan Yu, Qingru Zhang +4
Transformer models have achieved remarkable results in various natural language tasks, but they are often prohibitively large, requiring massive memories and computational resource…
Interactive Editing for Text Summarization
Yujia Xie, Xun Wang, Si-Qing Chen +2
Summarizing lengthy documents is a common and essential task in our daily lives. Although recent advancements in neural summarization models can assist in crafting general-purpose…
Summarization with Precise Length Control
Lesly Miculicich, Yujia Xie, Song Wang +1
Many applications of text generation such as summarization benefit from accurately controlling the text length. Existing approaches on length-controlled summarization either result…
Query Rewriting for Retrieval-Augmented Large Language Models
Xinbei Ma, Yeyun Gong, Pengcheng He +2
Large Language Models (LLMs) play powerful, black-box readers in the retrieve-then-read pipeline, making remarkable progress in knowledge-intensive tasks. This work introduces a ne…
PROM: A Phrase-level Copying Mechanism with Pre-training for Abstractive Summarization
Xinbei Ma, Yeyun Gong, Pengcheng He +2
Based on the remarkable achievements of pre-trained language models in abstractive summarization, the copying mechanism has proved helpful by improving the factuality, stability, a…
Personalized Abstractive Summarization by Tri-agent Generation Pipeline
Wen Xiao, Yujia Xie, Giuseppe Carenini +1
Tailoring outputs from large language models, like ChatGPT, to implicit user preferences remains a challenge despite their impressive generative capabilities. In this paper, we pro…
In-Context Learning Unlocked for Diffusion Models
Zhendong Wang, Yifan Jiang, Yadong Lu +5
We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models. Given a pair of task-specific example images, such as depth from/to…
POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models
Korawat Tanwisuth, Shujian Zhang, Huangjie Zheng +2
Through prompting, large-scale pre-trained models have become more expressive and powerful, gaining significant attention in recent years. Though these big models have zero-shot ca…
Instruction Tuning with GPT-4
Baolin Peng, Chunyuan Li, Pengcheng He +2
Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilitie…
Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
Zhendong Wang, Yifan Jiang, Huangjie Zheng +5
Diffusion models are powerful, but they require a lot of time and data to train. We propose Patch Diffusion, a generic patch-wise training framework, to significantly reduce the tr…
Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
Baolin Peng, Michel Galley, Pengcheng He +8
Large language models (LLMs), such as ChatGPT, are able to generate human-like, fluent responses for many downstream tasks, e.g., task-oriented dialog and question answering. Howev…
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Qingru Zhang, Minshuo Chen, Alexander Bukharin +5
Fine-tuning large pre-trained language models on downstream tasks has become an important paradigm in NLP. However, common practice fine-tunes all of the parameters in a pre-traine…
A Prototype-Oriented Clustering for Domain Shift with Source Privacy
Korawat Tanwisuth, Shujian Zhang, Pengcheng He +1
Unsupervised clustering under domain shift (UCDS) studies how to transfer the knowledge from abundant unlabeled data from multiple source domains to learn the representation of the…
Guiding Large Language Models via Directional Stimulus Prompting
Zekun Li, Baolin Peng, Pengcheng He +3
We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLM…
Attend to the Right Context: A Plug-and-Play Module for Content-Controllable Summarization
Wen Xiao, Lesly Miculicich, Yang Liu +2
Content-Controllable Summarization generates summaries focused on the given controlling signals. Due to the lack of large-scale training corpora for the task, we propose a plug-and…
Momentum Calibration for Text Generation
Xingxing Zhang, Yiran Liu, Xun Wang +5
The input and output of most text generation tasks can be transformed to two sequences of tokens and they can be modeled using sequence-to-sequence learning modeling tools such as…
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Yu Li, Baolin Peng, Pengcheng He +3
Dialogue summarization has recently garnered significant attention due to its wide range of applications. However, existing methods for summarizing dialogues have limitations becau…
HyperTuning: Toward Adapting Large Language Models without Back-propagation
Jason Phang, Yi Mao, Pengcheng He +1
Fine-tuning large language models for different tasks can be costly and inefficient, and even methods that reduce the number of tuned parameters still require full gradient-based o…
Less is More: Task-aware Layer-wise Distillation for Language Model Compression
Chen Liang, Simiao Zuo, Qingru Zhang +3
Layer-wise distillation is a powerful tool to compress large models (i.e. teacher models) into small ones (i.e., student models). The student distills knowledge from the teacher by…