5.7k citations · 8.2k across the 48 of their papers we have counts for
51 papers
COEVO: Co-Evolving Context and Parameters for Recursive Self-Improvement
Siwei Chen, Xinping Bao, Xinyu Cai +3
Recursive self-improvement (RSI) seeks to move large language models beyond static training pipelines toward systems that can participate in improving their own future behavior. Ex…
Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
Alexandre Galashov, Matt Jones, Rosemary Ke +3
Within the landscape of inference-time scaling methods for foundation models, a width-based approach to scaling -- which involves the insertion of <pause> tokens in the input strea…
Understanding the performance gap between online and offline alignment algorithms
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng +8
Reinforcement learning from human feedback (RLHF) is the canonical framework for large language model alignment. However, rising popularity in offline alignment algorithms challeng…
Retrieval Augmented End-to-End Spoken Dialog Models
Mingqiu Wang, Izhak Shafran, Hagen Soltau +4
We recently developed SLM, a joint speech and language model, which fuses a pretrained foundational speech model and a large language model (LLM), while preserving the in-context l…
IG Captioner: Information Gain Captioners are Strong Zero-shot Classifiers
Chenglin Yang, Siyuan Qiao, Yuan Cao +4
Generative training has been demonstrated to be powerful for building visual-language models. However, on zero-shot discriminative benchmarks, there is still a performance gap betw…
RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
Pierre Sermanet, Tianli Ding, Jeffrey Zhao +18
We present a scalable, bottom-up and intrinsically diverse data collection scheme that can be used for high-level reasoning with long and medium horizons and that has 2.2x higher t…
SLM: Bridge the thin gap between speech and text foundation models
Mingqiu Wang, Wei Han, Izhak Shafran +15
We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM…
Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding
Mingqiu Wang, Izhak Shafran, Hagen Soltau +4
Large Language Models (LLMs) have been applied in the speech domain, often incurring a performance drop due to misaligned between speech and language representations. To bridge thi…
Grammar Prompting for Domain-Specific Language Generation with Large Language Models
Bailin Wang, Zi Wang, Xuezhi Wang +3
Large language models (LLMs) can learn to perform a wide range of natural language tasks from just a handful of in-context examples. However, for generating strings from highly str…
Can Public Large Language Models Help Private Cross-device Federated Learning?
Boxin Wang, Yibo Jacky Zhang, Yuan Cao +5
We study (differentially) private federated learning (FL) of language models. The language models in cross-device FL are relatively small, which can be trained with meaningful form…
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Shunyu Yao, Dian Yu, Jeffrey Zhao +4
Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making proce…
Binarized Neural Machine Translation
Yichi Zhang, Ankush Garg, Yuan Cao +4
The rapid scaling of language models is motivating research using low-bitwidth quantization. In this work, we propose a novel binarization technique for Transformers applied to mac…
MUX-PLMs: Data Multiplexing for High-throughput Language Models
Vishvak Murahari, Ameet Deshpande, Carlos E. Jimenez +4
The widespread adoption of large language models such as ChatGPT and Bard has led to unprecedented demand for these technologies. The burgeoning cost of inference for ever-increasi…
Speech Aware Dialog System Technology Challenge (DSTC11)
Hagen Soltau, Izhak Shafran, Mingqiu Wang +6
Most research on task oriented dialog modeling is based on written text input. However, users interact with practical dialog systems often using speech as input. Typically, systems…
AnyTOD: A Programmable Task-Oriented Dialog System
Jeffrey Zhao, Yuan Cao, Raghav Gupta +6
We propose AnyTOD, an end-to-end, zero-shot task-oriented dialog (TOD) system capable of handling unseen tasks without task-specific training. We view TOD as a program executed by…
VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners
Shen Yan, Tao Zhu, Zirui Wang +5
We explore an efficient approach to establish a foundational video-text model. We present VideoCoCa that maximally reuses a pretrained image-text contrastive captioner (CoCa) model…
Knowledge-grounded Dialog State Tracking
Dian Yu, Mingqiu Wang, Yuan Cao +3
Knowledge (including structured knowledge such as schema and ontology, and unstructured knowledge such as web corpus) is a critical part of dialog understanding, especially for uns…
ReAct: Synergizing Reasoning and Acting in Language Models
Shunyu Yao, Jeffrey Zhao, Dian Yu +4
While large language models (LLMs) have demonstrated impressive capabilities across tasks in language understanding and interactive decision making, their abilities for reasoning (…
Unsupervised Slot Schema Induction for Task-oriented Dialog
Dian Yu, Mingqiu Wang, Yuan Cao +3
Carefully-designed schemas describing how to collect and annotate dialog corpora are a prerequisite towards building task-oriented dialog systems. In practical applications, manual…
The Implicit Length Bias of Label Smoothing on Beam Search Decoding
Bowen Liang, Pidong Wang, Yuan Cao
Label smoothing is ubiquitously applied in Neural Machine Translation (NMT) training. While label smoothing offers a desired regularization effect during model training, in this pa…
Building Machine Translation Systems for the Next Thousand Languages
Ankur Bapna, Isaac Caswell, Julia Kreutzer +21
In this paper we share findings from our effort to build practical machine translation (MT) systems capable of translating across over one thousand languages. We describe results i…
Show, Don't Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue
Raghav Gupta, Harrison Lee, Jeffrey Zhao +3
Building universal dialogue systems that operate across multiple domains/APIs and generalize to new ones with minimal overhead is a critical challenge. Recent works have leveraged…
Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation
Yong Cheng, Ankur Bapna, Orhan Firat +3
Multilingual neural machine translation models are trained to maximize the likelihood of a mix of examples drawn from multiple language pairs. The dominant inductive bias applied t…
Description-Driven Task-Oriented Dialog Modeling
Jeffrey Zhao, Raghav Gupta, Yuan Cao +6
Task-oriented dialogue (TOD) systems are required to identify key information from conversations for the completion of given tasks. Such information is conventionally specified in…