7 papers
RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech
Zhisheng Zheng, Xiaohang Sun, Tuan Dinh +8
End-to-end speech-to-speech translation (S2ST) systems typically struggle with a critical data bottleneck: the scarcity of parallel speech-to-speech corpora. To overcome this, we i…
VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editing
Zhisheng Zheng, Puyuan Peng, Anuj Diwan +5
We introduce VoiceCraft-X, an autoregressive neural codec language model which unifies multilingual speech editing and zero-shot Text-to-Speech (TTS) synthesis across 11 languages:…
RefTok: Reference-Based Tokenization for Video Generation
Xiang Fan, Xiaohang Sun, Kushan Thakkar +4
Effectively handling temporal redundancy remains a key challenge in learning video models. Prevailing approaches often treat each set of frames independently, failing to effectivel…
CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation
Guofeng Cui, Pichao Wang, Yang Liu +3
Large language models (LLMs) have shown great potential in natural language processing tasks, but their application to machine translation (MT) remains challenging due to pretraini…
Beyond Speaker Identity: Text Guided Target Speech Extraction
Mingyue Huo, Abhinav Jain, Cong Phuoc Huynh +4
Target Speech Extraction (TSE) traditionally relies on explicit clues about the speaker's identity like enrollment audio, face images, or videos, which may not always be available.…
GEXIA: Granularity Expansion and Iterative Approximation for Scalable Multi-grained Video-language Learning
Yicheng Wang, Zhikang Zhang, Jue Wang +6
In various video-language learning tasks, the challenge of achieving cross-modality alignment with multi-grained data persists. We propose a method to tackle this challenge from tw…