19 citations · 28 across the 16 of their papers we have counts for
16 papers
FlowPortrait: Reinforcement Learning for Audio-Driven Portrait Video Generation
Weiting Tan, Andy T. Liu, Ming Tu +3
Generating realistic talking-head videos remains challenging due to persistent issues such as imperfect lip synchronization, unnatural motion, and evaluation metrics that correlate…
Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents
Weiting Tan, Xinghua Qu, Ming Tu +4
Effective interactive tool use requires agents to master Tool Integrated Reasoning (TIR): a complex process involving multi-turn planning and long-context dialogue management. To t…
Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets
Tianjian Li, Haoran Xu, Weiting Tan +2
Data abundance across different domains exhibits a long-tailed distribution: few domains have abundant data, while most face data scarcity. Our work focuses on a multilingual setti…
SSR: Alignment-Aware Modality Connector for Speech Language Models
Weiting Tan, Hirofumi Inaguma, Ning Dong +2
Fusing speech into pre-trained language model (SpeechLM) usually suffers from inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We…
It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF
Taiming Lu, Lingfeng Shen, Xinyu Yang +3
Reinforcement Learning from Human Feedback (RLHF) involves training policy models (PMs) and reward models (RMs) to align language models with human preferences. Instead of focusing…
DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation
Weiting Tan, Jingyu Zhang, Lingfeng Shen +2
Non-autoregressive Transformers (NATs) are recently applied in direct speech-to-speech translation systems, which convert speech across different languages without intermediate tex…
Streaming Sequence Transduction through Dynamic Compression
Weiting Tan, Yunmo Chen, Tongfei Chen +5
We introduce STAR (Stream Transduction with Anchor Representations), a novel Transformer-based model designed for efficient sequence-to-sequence transduction over streams. STAR dyn…
The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts
Lingfeng Shen, Weiting Tan, Sihao Chen +6
As the influence of large language models (LLMs) spans across global communities, their safety challenges in multilingual settings become paramount for alignment research. This pap…
Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
Haoran Xu, Amr Sharaf, Yunmo Chen +5
Moderate-sized large language models (LLMs) -- those with 7B or 13B parameters -- exhibit promising machine translation (MT) performance. However, even the top-performing 13B LLM-b…
Structure-Aware Path Inference for Neural Finite State Transducers
Weiting Tan, Chu-cheng Lin, Jason Eisner
Neural finite-state transducers (NFSTs) form an expressive family of neurosymbolic sequence transduction models. An NFST models each string pair as having been generated by a laten…
Narrowing the Gap between Zero- and Few-shot Machine Translation by Matching Styles
Weiting Tan, Haoran Xu, Lingfeng Shen +5
Large language models trained primarily in a monolingual setting have demonstrated their ability to generalize to machine translation using zero- and few-shot examples with in-cont…
Condensing Multilingual Knowledge with Lightweight Language-Specific Modules
Haoran Xu, Weiting Tan, Shuyue Stella Li +4
Incorporating language-specific (LS) modules is a proven method to boost performance in multilingual machine translation. This approach bears similarity to Mixture-of-Experts (MoE)…
Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiency
Lingfeng Shen, Weiting Tan, Boyuan Zheng +1
With growing capabilities of large language models, prompting them has become the dominant way to access them. This has motivated the development of strategies for automatically se…
Multilingual Representation Distillation with Contrastive Learning
Weiting Tan, Kevin Heffernan, Holger Schwenk +1
Multilingual sentence representations from large models encode semantic information from two or more languages and can be used for different cross-lingual information retrieval and…
Bitext Mining for Low-Resource Languages via Contrastive Learning
Weiting Tan, Philipp Koehn
Mining high-quality bitexts for low-resource languages is challenging. This paper shows that sentence representation of language models fine-tuned with multiple negatives ranking l…
Doubly-Trained Adversarial Data Augmentation for Neural Machine Translation
Weiting Tan, Shuoyang Ding, Huda Khayrallah +1
Neural Machine Translation (NMT) models are known to suffer from noisy inputs. To make models robust, we generate adversarial augmentation samples that attack the model and preserv…