Publications (334)
On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation
Liang Chen, Shuming Ma, Dongdong Zhang +2
Task-Specific Expert Pruning for Sparse Mixture-of-Experts
Tianyu Chen, Shaohan Huang, Yuan Xie +5
Can Monolingual Pretrained Models Help Cross-Lingual Classification?
Zewen Chi, Li Dong, Furu Wei +2
Multi-Document Summarization via Discriminative Summary Reranking
Xiaojun Wan, Ziqiang Cao, Furu Wei +2
Black-Box On-Policy Distillation of Large Language Models
Tianzhu Ye, Li Dong, Zewen Chi +3
Controllable Natural Language Generation with Contrastive Prefixes
Jing Qian, Li Dong, Yelong Shen +2
Retentive Network: A Successor to Transformer for Large Language Models
Yutao Sun, Li Dong, Shaohan Huang +5
xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
Xin Cheng, Xun Wang, Xingxing Zhang +5
Online Experiential Learning for Language Models
Tianzhu Ye, Li Dong, Qingxiu Dong +3
UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
Sanyuan Chen, Yu Wu, Chengyi Wang +8
Visualizing and Understanding the Effectiveness of BERT
Yaru Hao, Li Dong, Furu Wei +1
Computer Environments Elicit General Agentic Intelligence in LLMs
Daixuan Cheng, Shaohan Huang, Yuxian Gu +6
Thinking Augmented Pre-training
Liang Wang, Nan Yang, Shaohan Huang +2
Scheduled DropHead: A Regularization Method for Transformer Models
Wangchunshu Zhou, Tao Ge, Ke Xu +2
Examining False Positives under Inference Scaling for Mathematical Reasoning
Yu Wang, Nan Yang, Liang Wang +2
Improving Domain Adaptation through Extended-Text Reading Comprehension
Ting Jiang, Shaohan Huang, Shengyue Luo +8
Improving Non-autoregressive Generation with Mixup Training
Ting Jiang, Shaohan Huang, Zihan Zhang +6
Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
Di Zhang, Xun Wu, Shaohan Huang +9
K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoning
Yadong Zhang, Shaoguang Mao, Tao Ge +4
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
Tianyu Chen, Hangbo Bao, Shaohan Huang +6
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Yunzhi Yao, Shaohan Huang, Wenhui Wang +2
SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data
Ziqiang Zhang, Sanyuan Chen, Long Zhou +8
Geometric-Mean Policy Optimization
Yuzhong Zhao, Yue Liu, Junpeng Liu +9
Language Is Not All You Need: Aligning Perception with Language Models
Shaohan Huang, Li Dong, Wenhui Wang +15
Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization
Ruipeng Jia, Xingxing Zhang, Yanan Cao +3
GeAR: Generation Augmented Retrieval
Haoyu Liu, Shaohan Huang, Jianfeng Liu +6
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Liang Wang, Nan Yang, Xiaolong Huang +5
Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
Chengzu Li, Wenshan Wu, Huanyu Zhang +5
Improving Grammatical Error Correction with Machine Translation Pairs
Wangchunshu Zhou, Tao Ge, Chang Mu +3
Attention-Guided Answer Distillation for Machine Reading Comprehension
Minghao Hu, Yuxing Peng, Furu Wei +4
Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
Hongyu Wang, Shuming Ma, Ruiping Wang +1
Are More Layers Beneficial to Graph Transformers?
Haiteng Zhao, Shuming Ma, Dongdong Zhang +2
HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition
Yuxuan Liu, Tianchi Yang, Shaohan Huang +6
Faithful to the Original: Fact Aware Neural Abstractive Summarization
Ziqiang Cao, Furu Wei, Wenjie Li +1
TGSum: Build Tweet Guided Multi-Document Summarization Dataset
Ziqiang Cao, Chengyao Chen, Wenjie Li +3
Swin Transformer V2: Scaling Up Capacity and Resolution
Ze Liu, Han Hu, Yutong Lin +9
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Sanyuan Chen, Chengyi Wang, Zhengyang Chen +15
UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark
Yanlin Li, Minghui Guo, Kaiwen Zhang +13
Neural Melody Composition from Lyrics
Hangbo Bao, Shaohan Huang, Furu Wei +5
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
Heming Xia, Tao Ge, Peiyi Wang +3
Towards Optimal Learning of Language Models
Yuxian Gu, Li Dong, Yaru Hao +3
Multilingual E5 Text Embeddings: A Technical Report
Liang Wang, Nan Yang, Xiaolong Huang +3
On the Representation Collapse of Sparse Mixture of Experts
Zewen Chi, Li Dong, Shaohan Huang +9
s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Hangbo Bao, Li Dong, Wenhui Wang +2
Preference Optimization for Reasoning with Pseudo Feedback
Fangkai Jiao, Geyang Guo, Xingxing Zhang +3
VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
Tengchao Lv, Lei Cui, Momcilo Vasilijevic +1
A Length-Extrapolatable Transformer
Yutao Sun, Li Dong, Barun Patra +6
Democratizing Reasoning Ability: Tailored Learning from Large Language Model
Zhaoyang Wang, Shaohan Huang, Yuxuan Liu +8
BEiT: BERT Pre-Training of Image Transformers
Hangbo Bao, Li Dong, Songhao Piao +1
BERT Loses Patience: Fast and Robust Inference with Early Exit
Wangchunshu Zhou, Canwen Xu, Tao Ge +3
LayoutLM: Pre-training of Text and Layout for Document Image Understanding
Yiheng Xu, Minghao Li, Lei Cui +3
The Era of Agentic Organization: Learning to Organize with Language Models
Zewen Chi, Li Dong, Qingxiu Dong +4
Multi-Head Mixture-of-Experts
Xun Wu, Shaohan Huang, Wenhui Wang +1
Consistency Regularization for Cross-Lingual Fine-Tuning
Bo Zheng, Li Dong, Shaohan Huang +7
Generic-to-Specific Distillation of Masked Autoencoders
Wei Huang, Zhiliang Peng, Li Dong +3
Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation
Kun Wei, Long Zhou, Ziqiang Zhang +5
EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation
Tao Ge, Si-Qing Chen, Furu Wei
Autoregressive Speech Synthesis without Vector Quantization
Lingwei Meng, Long Zhou, Shujie Liu +9
Synergizing Understanding and Generation with Interleaved Analyzing-Drafting Thinking
Shengqiong Wu, Bobo Li, Xinkai Wang +6
Semi-Parametric Retrieval via Binary Bag-of-Tokens Index
Jiawei Zhou, Li Dong, Furu Wei +1
Neural Question Generation from Text: A Preliminary Study
Qingyu Zhou, Nan Yang, Furu Wei +3
TorchScale: Transformers at Scale
Shuming Ma, Hongyu Wang, Shaohan Huang +8
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Zewen Chi, Li Dong, Bo Zheng +4
Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
Damai Dai, Yutao Sun, Li Dong +4
ShifCon: Enhancing Non-Dominant Language Capabilities with a Shift-based Multilingual Contrastive Framework
Hengyuan Zhang, Chenming Shang, Sizhe Wang +6
Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Models
Hongyuan Lu, Haoyang Huang, Shuming Ma +3
Augmenting Language Models with Long-Term Memory
Weizhi Wang, Li Dong, Hao Cheng +4
Reward Reasoning Model
Jiaxin Guo, Zewen Chi, Li Dong +4
VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
Qiushi Zhu, Long Zhou, Ziqiang Zhang +7
Transforming Wikipedia into Augmented Data for Query-Focused Summarization
Haichao Zhu, Li Dong, Furu Wei +2
Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
Guanhua Chen, Shuming Ma, Yun Chen +5
Selective Encoding for Abstractive Sentence Summarization
Qingyu Zhou, Nan Yang, Furu Wei +1
DocReward: A Document Reward Model for Structuring and Stylizing
Junpeng Liu, Yuzhong Zhao, Bowen Cao +17
Next Block Prediction: Video Generation via Semi-Autoregressive Modeling
Shuhuai Ren, Shuming Ma, Xu Sun +1
CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation
Jian Yang, Shaohan Huang, Shuming Ma +6
Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training
Chengyi Wang, Yiming Wang, Yu Wu +4
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
Yaru Hao, Li Dong, Furu Wei +1
Fine-Tuning LLaMA for Multi-Stage Text Retrieval
Xueguang Ma, Liang Wang, Nan Yang +2
Improving Text Embeddings with Large Language Models
Liang Wang, Nan Yang, Xiaolong Huang +3
Data Selection via Optimal Control for Language Models
Yuxian Gu, Li Dong, Hongning Wang +4
Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Wangyou Zhang, Zhuo Chen, Naoyuki Kanda +8
Distilled Dual-Encoder Model for Vision-Language Understanding
Zekun Wang, Wenhui Wang, Haichao Zhu +3
Reasoning with Exploration: An Entropy Perspective
Daixuan Cheng, Shaohan Huang, Xuekai Zhu +4
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Zewen Chi, Li Dong, Furu Wei +7
A Unified View of Masked Image Modeling
Zhiliang Peng, Li Dong, Hangbo Bao +2
Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?
Sanyuan Chen, Yu Wu, Chengyi Wang +8
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
Bo Zheng, Li Dong, Shaohan Huang +5
Two Pathways to Truthfulness: On the Intrinsic Encoding of LLM Hallucinations
Wen Luo, Guangyue Peng, Wei Li +8
VIBEVOICE-ASR Technical Report
Zhiliang Peng, Jianwei Yu, Yaoyao Chang +21
Scaling Data Difficulty: Improving Coding Models via Reinforcement Learning on Fresh and Challenging Problems
Zongqian Li, Tengchao Lv, Shaohan Huang +8
MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Zewen Chi, Li Dong, Shuming Ma +3
Jointly Learning to Repair Code and Generate Commit Message
Jiaqi Bai, Long Zhou, Ambrosio Blanco +4
ResLoRA: Identity Residual Mapping in Low-Rank Adaption
Shuhua Shi, Shaohan Huang, Minghui Song +7
Optimizing Prompts for Text-to-Image Generation
Yaru Hao, Zewen Chi, Li Dong +1
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Shuming Ma, Jian Yang, Haoyang Huang +10
BitNet Distillation
Xun Wu, Shaohan Huang, Wenhui Wang +4
Context-DPO: Aligning Language Models for Context-Faithfulness
Baolong Bi, Shaohan Huang, Yiwei Wang +11
Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References
Tianyi Tang, Hongyuan Lu, Yuchen Eleanor Jiang +5
Prototypical Calibration for Few-shot Learning of Language Models
Zhixiong Han, Yaru Hao, Li Dong +2
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
Shuming Ma, Li Dong, Shaohan Huang +6