1.6k citations · 4.2k across the 75 of their papers we have counts for
25 papers · 1 filter
Scaling Properties of Continuous Diffusion Spoken Language Models
Jason Ramapuram, Eeshan Gunesh Dhekane, Amitis Shidani +6
Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance, with recent discrete autoregressive (AR) SLMs indicating significant computational…
Embarrassingly Simple Self-Distillation Improves Code Generation
Ruixiang Zhang, Richard He Bai, Huangjie Zheng +3
Can a large language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirma…
CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning
Jie He, Richard He Bai, Sinead Williamson +3
Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge but still suffers from long contexts and disjoint retrieval-generation optimizati…
Closing the Gap Between Text and Speech Understanding in LLMs
Santiago Cuervo, Skyler Seto, Maureen de Seyssel +5
Large Language Models (LLMs) can be adapted to extend their text capabilities to speech inputs. However, these speech-adapted LLMs consistently underperform their text-based counte…
Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition
Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly
Mixture-of-experts (MoE) architectures have expanded from language modeling to automatic speech recognition (ASR). Traditional MoE methods, such as the Switch Transformer, route ex…
DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation
Shansan Gong, Ruixiang Zhang, Huangjie Zheng +4
Diffusion large language models (dLLMs) are compelling alternatives to autoregressive (AR) models because their denoising models operate over the entire sequence. The global planni…
SpeakStream: Streaming Text-to-Speech with Interleaved Data
Richard He Bai, Zijin Gu, Tatiana Likhomanenko +1
The latency bottleneck of traditional text-to-speech (TTS) systems fundamentally hinders the potential of streaming large language models (LLMs) in conversational AI. These TTS sys…
SAGE: Steering Dialog Generation with Future-Aware State-Action Augmentation
Yizhe Zhang, Navdeep Jaitly
Recent advances in large language models have demonstrated impressive capabilities in task-oriented applications, yet building emotionally intelligent chatbots that can engage in n…
What Makes the Preferred Thinking Direction for LLMs in Multiple-choice Questions?
Yizhe Zhang, Richard Bai, Zijin Gu +5
Language models usually use left-to-right (L2R) autoregressive factorization. However, L2R factorization may not always be the best inductive bias. Therefore, we investigate whethe…
Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models
Yifu Qiu, Varun Embar, Yizhe Zhang +3
Recent advancements in long-context language models (LCLMs) promise to transform Retrieval-Augmented Generation (RAG) by simplifying pipelines. With their expanded context windows,…
dMel: Speech Tokenization made Simple
Richard He Bai, Tatiana Likhomanenko, Ruixiang Zhang +3
Large language models have revolutionized natural language processing by leveraging self-supervised pretraining on vast textual data. Inspired by this success, researchers have inv…
Divide-or-Conquer? Which Part Should You Distill Your LLM?
Zhuofeng Wu, He Bai, Aonan Zhang +4
Recent methods have demonstrated that Large Language Models (LLMs) can solve reasoning tasks better when they are encouraged to solve subtasks of the main task first. In this paper…
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
Pratyush Maini, Skyler Seto, He Bai +3
Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data req…
Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games
Yizhe Zhang, Jiarui Lu, Navdeep Jaitly
Large language models (LLMs) are effective at answering questions that are clearly asked. However, when faced with ambiguous queries they can act unpredictably and produce incorrec…
Construction of Paired Knowledge Graph-Text Datasets Informed by Cyclic Evaluation
Ali Mousavi, Xin Zhan, He Bai +9
Datasets that pair Knowledge Graphs (KG) and text together (KG-T) can be used to train forward and reverse neural models that generate text from KG and vice versa. However models t…
PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model
Yizhe Zhang, Jiatao Gu, Zhuofeng Wu +3
Autoregressive models for text sometimes generate repetitive and low-quality output because errors accumulate during the steps of generation. This issue is often attributed to expo…
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
Jonathan Shen, Ruoming Pang, Ron J. Weiss +10
This paper describes Tacotron 2, a neural network architecture for speech synthesis directly from text. The system is composed of a recurrent sequence-to-sequence feature predictio…
State-of-the-art Speech Recognition With Sequence-to-Sequence Models
Chung-Cheng Chiu, Tara N. Sainath, Yonghui Wu +11
Attention-based encoder-decoder architectures such as Listen, Attend, and Spell (LAS), subsume the acoustic, pronunciation and language model components of a traditional automatic…
Speech recognition for medical conversations
Chung-Cheng Chiu, Anshuman Tripathi, Katherine Chou +11
In this work we explored building automatic speech recognition models for transcribing doctor patient conversation. We collected a large scale dataset of clinical conversations ($1…
An online sequence-to-sequence model for noisy speech recognition
Chung-Cheng Chiu, Dieterich Lawson, Yuping Luo +4
Generative models have long been the dominant approach for speech recognition. The success of these models however relies on the use of sophisticated recipes and complicated machin…
Tacotron: Towards End-to-End Speech Synthesis
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton +11
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module. Building these component…
Sequence-to-Sequence Models Can Directly Translate Foreign Speech
Ron J. Weiss, Jan Chorowski, Navdeep Jaitly +2
We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcri…
RNN Approaches to Text Normalization: A Challenge
Richard Sproat, Navdeep Jaitly
This paper presents a challenge to the community: given a large corpus of written text aligned to its normalized spoken form, train an RNN to learn the correct normalization functi…
Very Deep Convolutional Networks for End-to-End Speech Recognition
Yu Zhang, William Chan, Navdeep Jaitly
Sequence-to-sequence models have shown success in end-to-end speech recognition. However these models have only used shallow acoustic encoder networks. In our work, we successively…