241 citations · 334 across the 19 of their papers we have counts for
19 papers
Investigating Decoder-only Large Language Models for Speech-to-text Translation
Chao-Wei Huang, Hui Lu, Hongyu Gong +4
Large language models (LLMs), known for their exceptional reasoning capabilities, generalizability, and fluency across diverse domains, present a promising avenue for enhancing spe…
Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation
Min-Jae Hwang, Ilia Kulikov, Benjamin Peloquin +3
In this paper, we propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-to-speech translation (S2ST). Recently proposed…
MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation
Yifan Peng, Ilia Kulikov, Yilin Yang +4
There have been emerging research interest and advances in speech-to-speech translation (S2ST), translating utterances from one language to another. This work proposes Multitask Sp…
An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesis
Yifan Peng, Ilia Kulikov, Yilin Yang +4
Speech language models (LMs) are promising for high-quality speech synthesis through in-context learning. A typical speech LM takes discrete semantic units as content and a short u…
Seamless: Multilingual Expressive and Streaming Speech Translation
Seamless Communication, Loïc Barrault, Yu-An Chung +62
Large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this wo…
Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction
Jiatong Shi, Hirofumi Inaguma, Xutai Ma +2
Existing Self-Supervised Learning (SSL) models for speech typically process speech signals at a fixed resolution of 20 milliseconds. This approach overlooks the varying information…
SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
Seamless Communication, Loïc Barrault, Yu-An Chung +65
What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages? While recent breakthroughs in text-based models have pushed…
UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units
Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov +7
Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified p…
Improving Speech-to-Speech Translation Through Unlabeled Text
Xuan-Phi Nguyen, Sravya Popuri, Changhan Wang +3
Direct speech-to-speech translation (S2ST) is among the most challenging problems in the translation paradigm due to the significant scarcity of S2ST data. While effort has been ma…
Simple and Effective Unsupervised Speech Translation
Changhan Wang, Hirofumi Inaguma, Peng-Jen Chen +5
The amount of labeled data to train models for speech tasks is limited for most languages, however, the data scarcity is exacerbated for speech translation which requires labeled d…
Named Entity Detection and Injection for Direct Speech Translation
Marco Gaido, Yun Tang, Ilia Kulikov +3
In a sentence, certain words are critical for its semantic. Among them, named entities (NEs) are notoriously challenging for neural models. Despite their importance, their accurate…
Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Models
Felix Stahlberg, Ilia Kulikov, Shankar Kumar
In many natural language processing (NLP) tasks the same input (e.g. source sentence) can have multiple possible outputs (e.g. translations). To analyze how this ambiguity (also kn…
Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling
Ilia Kulikov, Maksim Eremeev, Kyunghyun Cho
Neural autoregressive sequence models smear the probability among many possible sequences including degenerate ones, such as empty or repetitive sequences. In this work, we tackle…
Mode recovery in neural autoregressive sequence modeling
Ilia Kulikov, Sean Welleck, Kyunghyun Cho
Despite its wide use, recent studies have revealed unexpected and undesirable properties of neural autoregressive sequence models trained with maximum likelihood, such as an unreas…
Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
Sean Welleck, Ilia Kulikov, Jaedeok Kim +2
Despite strong performance on a variety of tasks, neural sequence models trained with maximum likelihood have been shown to exhibit issues such as length bias and degenerate repeti…
Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
Margaret Li, Stephen Roller, Ilia Kulikov +4
Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not address. They tend to produce generations that (i) rely to…
Neural Text Generation with Unlikelihood Training
Sean Welleck, Ilia Kulikov, Stephen Roller +3
Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and de…
Multi-Turn Beam Search for Neural Dialogue Modeling
Ilia Kulikov, Jason Lee, Kyunghyun Cho
In neural dialogue modeling, a neural network is trained to predict the next utterance, and at inference time, an approximate decoding algorithm is used to generate next utterances…
Importance of Search and Evaluation Strategies in Neural Dialogue Modeling
Ilia Kulikov, Alexander H. Miller, Kyunghyun Cho +1
We investigate the impact of search strategies in neural dialogue modeling. We first compare two standard search algorithms, greedy and beam search, as well as our newly proposed i…