45 citations · 301 across the 53 of their papers we have counts for
55 papers
Speech Translation with Large Language Models: An Industrial Practice
Zhichao Huang, Rong Ye, Tom Ko +4
Given the great success of large language models (LLMs) across various tasks, in this paper, we introduce LLM-ST, a novel and effective speech translation model constructed upon a…
BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training
Yiming Yan, Tao Wang, Chengqi Zhao +3
Automatic metrics play a crucial role in machine translation. Despite the widespread use of n-gram-based metrics, there has been a recent surge in the development of pre-trained mo…
Recent Advances in Direct Speech-to-text Translation
Chen Xu, Rong Ye, Qianqian Dong +5
Recently, speech-to-text translation has attracted more and more attention and many studies have emerged rapidly. In this paper, we present a comprehensive survey on direct speech…
MOSPC: MOS Prediction Based on Pairwise Comparison
Kexin Wang, Yunlong Zhao, Qianqian Dong +2
As a subjective metric to evaluate the quality of synthesized speech, Mean opinion score~(MOS) usually requires multiple annotators to score the same speech. Such an annotation app…
Understanding Parameter Sharing in Transformers
Ye Lin, Mingxuan Wang, Zhexi Zhang +3
Parameter sharing has proven to be a parameter-efficient approach. Previous work on Transformers has focused on sharing parameters in different layers, which can improve the perfor…
PolyVoice: Language Models for Speech to Speech Translation
Qianqian Dong, Zhiying Huang, Qiao Tian +15
We propose PolyVoice, a language model-based framework for speech-to-speech translation (S2ST) system. Our framework consists of two language models: a translation language model a…
MobileNMT: Enabling Translation in 15MB and 30ms
Ye Lin, Xiaohui Wang, Zhexi Zhang +3
Deploying NMT models on mobile devices is essential for privacy, low latency, and offline scenarios. For high model capacity, NMT models are rather large. Running these models on d…
CTC-based Non-autoregressive Speech Translation
Chen Xu, Xiaoqian Liu, Xiaowen Liu +9
Combining end-to-end speech translation (ST) and non-autoregressive (NAR) generation is promising in language and speech processing for their advantages of less error propagation a…
DUB: Discrete Unit Back-translation for Speech Translation
Dong Zhang, Rong Ye, Tom Ko +2
How can speech-to-text translation (ST) perform as well as machine translation (MT)? The key point is to bridge the modality gap between speech and text so that useful MT technique…
BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation
Liyan Kang, Luyang Huang, Ningxin Peng +6
We present a large-scale video subtitle translation dataset, BigVideo, to facilitate the study of multi-modality machine translation. Compared with the widely used How2 and VaTeX d…
DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises
Jiasheng Ye, Zaixiang Zheng, Yu Bao +2
While diffusion models have achieved great success in generating continuous signals such as images and audio, it remains elusive for diffusion models in learning discrete sequence…
M3ST: Mix at Three Levels for Speech Translation
Xuxin Cheng, Qianqian Dong, Fengpeng Yue +3
How to solve the data scarcity problem for end-to-end speech-to-text translation (ST)? It's well known that data augmentation is an efficient method to improve performance for many…
Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation
Yaoming Zhu, Zewei Sun, Shanbo Cheng +3
Multimodal machine translation (MMT) aims to improve translation quality by incorporating information from other modalities, such as vision. Previous MMT systems mainly focus on be…
SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes
Wenda Xu, Xian Qian, Mingxuan Wang +2
Is it possible to train a general metric for evaluating text generation quality without human annotated ratings? Existing learned metrics either perform unsatisfactorily across tex…
Leveraging per Image-Token Consistency for Vision-Language Pre-training
Yunhao Gou, Tom Ko, Hansi Yang +3
Most existing vision-language pre-training (VLP) approaches adopt cross-modal masked language modeling (CMLM) to learn vision-language associations. However, we find that CMLM is i…
PARAGEN : A Parallel Generation Toolkit
Jiangtao Feng, Yi Zhou, Jun Zhang +7
PARAGEN is a PyTorch-based NLP toolkit for further development on parallel generation. PARAGEN provides thirteen types of customizable plugins, helping users to experiment quickly…
CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning
Chutong Meng, Junyi Ao, Tom Ko +2
Speech is the surface form of a finite set of phonetic units, which can be represented by discrete codes. We propose the Code BERT (CoBERT) approach for self-supervised speech repr…
Zero-shot Domain Adaptation for Neural Machine Translation with Retrieved Phrase-level Prompts
Zewei Sun, Qingnan Jiang, Shujian Huang +3
Domain adaptation is an important challenge for neural machine translation. However, the traditional fine-tuning solution requires multiple extra training and yields a high cost. I…
Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation
Qianqian Dong, Fengpeng Yue, Tom Ko +3
Direct Speech-to-speech translation (S2ST) has drawn more and more attention recently. The task is very challenging due to data scarcity and complex speech-to-speech mapping. In th…
Cross-modal Contrastive Learning for Speech Translation
Rong Ye, Mingxuan Wang, Lei Li
How can we learn unified representations for spoken utterances and their written text? Learning similar representations for semantically similar speech and text is important for sp…
GigaST: A 10,000-hour Pseudo Speech Translation Corpus
Rong Ye, Chengqi Zhao, Tom Ko +4
This paper introduces GigaST, a large-scale pseudo speech translation (ST) corpus. We create the corpus by translating the text in GigaSpeech, an English ASR corpus, into German an…
STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation
Qingkai Fang, Rong Ye, Lei Li +2
How to learn a better speech representation for end-to-end speech-to-text translation (ST) with limited labeled data? Existing techniques often attempt to transfer powerful machine…
Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus
Yaoming Zhu, Liwei Wu, Shanbo Cheng +1
The punctuation restoration task aims to correctly punctuate the output transcriptions of automatic speech recognition systems. Previous punctuation models, either using text only…
LightSeq2: Accelerated Training for Transformer-based Models on GPUs
Xiaohui Wang, Yang Wei, Ying Xiong +5
Transformer-based neural models are used in many AI applications. Training these models is expensive, as it takes huge GPU resources and long duration. It is challenging because ty…