Publications (140)
Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning
Yiming Huang, Xiao Liu, Yeyun Gong +4
DSCOVR: Randomized Primal-Dual Block Coordinate Algorithms for Asynchronous Distributed Optimization
Lin Xiao, Adams Wei Yu, Qihang Lin +1
GRIN: GRadient-INformed MoE
Liyuan Liu, Young Jin Kim, Shuohang Wang +14
HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization
Jiaao Chen, Dinghan Shen, Weizhu Chen +1
MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
Yaming Yang, Dilxat Muhtar, Yelong Shen +9
Adversarial Training for Large Neural Language Models
Xiaodong Liu, Hao Cheng, Pengcheng He +4
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Zhibin Gou, Zhihong Shao, Yeyun Gong +4
Controllable Natural Language Generation with Contrastive Prefixes
Jing Qian, Li Dong, Yelong Shen +2
Poolingformer: Long Document Modeling with Pooling Attention
Hang Zhang, Yeyun Gong, Yelong Shen +4
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
Yixiao Li, Yifan Yu, Chen Liang +4
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis +5
What Makes Good In-Context Examples for GPT-$3$?
Jiachang Liu, Dinghan Shen, Yizhe Zhang +3
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
Microsoft, :, Abdelrahman Abouelenin +73
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Pengcheng He, Xiaodong Liu, Jianfeng Gao +1
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models
Shengnan An, Yifei Li, Zeqi Lin +6
AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
Xingwei He, Zhenghao Lin, Yeyun Gong +7
ARCH: Efficient Adversarial Regularized Training with Caching
Simiao Zuo, Chen Liang, Haoming Jiang +5
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Tianyu Liu, Yizhe Zhang, Chris Brockett +4
Test-time Recursive Thinking: Self-Improvement without External Feedback
Yufan Zhuang, Chandan Singh, Liyuan Liu +5
Competition-Level Problems are Effective LLM Evaluators
Yiming Huang, Zhenghao Lin, Xiao Liu +8
A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
Dinghan Shen, Mingzhi Zheng, Yelong Shen +2
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
Chen Liang, Simiao Zuo, Minshuo Chen +5
CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing
Chen Liang, Pengcheng He, Yelong Shen +2
Rethinking Language Model Scaling under Transferable Hypersphere Optimization
Liliang Ren, Yang Liu, Yelong Shen +1
Exploring the Mystery of Influential Data for Mathematical Reasoning
Xinzhe Ni, Yeyun Gong, Zhibin Gou +4
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
Yixiao Li, Yifan Yu, Qingru Zhang +4
Meet in the Middle: A New Pre-training Paradigm
Anh Nguyen, Nikos Karampatziakis, Weizhu Chen
Conditional Self-Attention for Query-based Summarization
Yujia Xie, Tianyi Zhou, Yi Mao +1
ThetaEvolve: Test-time Learning on Open Problems
Yiping Wang, Shao-Rong Su, Zhiyuan Zeng +13
On the Variance of the Adaptive Learning Rate and Beyond
Liyuan Liu, Haoming Jiang, Pengcheng He +4
Learning From Mistakes Makes LLM Better Reasoner
Shengnan An, Zexiong Ma, Zeqi Lin +3
Less is More: Task-aware Layer-wise Distillation for Language Model Compression
Chen Liang, Simiao Zuo, Qingru Zhang +3
Parameter-free Sentence Embedding via Orthogonal Basis
Ziyi Yang, Chenguang Zhu, Weizhu Chen
COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs
Liming Liu, Zhenghao Xu, Zixuan Zhang +5
Automatic Instruction Evolving for Large Language Models
Weihao Zeng, Can Xu, Yingxiu Zhao +2
A Note on LoRA
Vlad Fomenko, Han Yu, Jongho Lee +2
A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation
Weizhen Qi, Yeyun Gong, Yelong Shen +6
R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science
Xu Yang, Xiao Yang, Shikai Fang +13
LLMs Can Generate a Better Answer by Aggregating Their Own Responses
Zichong Li, Xinyu Feng, Yuheng Cai +6
RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Fengji Zhang, Bei Chen, Yue Zhang +6
Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arena
Haipeng Luo, Qingfeng Sun, Can Xu +6
A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models
Woojeong Jin, Yu Cheng, Yelong Shen +2
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
Liliang Ren, Yang Liu, Yadong Lu +3
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
Pengcheng He, Jianfeng Gao, Weizhu Chen
Memory-Efficient Differentiable Transformer Architecture Search
Yuekai Zhao, Li Dong, Yelong Shen +3
X-SQL: reinforce schema representation with context
Pengcheng He, Yi Mao, Kaushik Chakrabarti +1
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
Zhihong Shao, Yeyun Gong, Yelong Shen +3
Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
Mingzhi Zheng, Dinghan Shen, Yelong Shen +2
HyperTuning: Toward Adapting Large Language Models without Back-propagation
Jason Phang, Yi Mao, Pengcheng He +1
Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
Yueqin Yin, Zhendong Wang, Yi Gu +3
StreamAdapter: Efficient Test Time Adaptation from Contextual Streams
Dilxat Muhtar, Yelong Shen, Yaming Yang +11
Rho-1: Not All Tokens Are What You Need
Zhenghao Lin, Zhibin Gou, Yeyun Gong +8
Language Models can be Logical Solvers
Jiazhan Feng, Ruochen Xu, Junheng Hao +4
Rider: Reader-Guided Passage Reranking for Open-Domain Question Answering
Yuning Mao, Pengcheng He, Xiaodong Liu +4
Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs
Huangjie Zheng, Pengcheng He, Weizhu Chen +1
AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Tong Wu, Zhihao Fan, Xiao Liu +9
Gold-Medal-Level Olympiad Geometry Solving with Efficient Heuristic Auxiliary Constructions
Boyan Duan, Xiao Liang, Shuai Lu +7
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
Haoming Jiang, Pengcheng He, Weizhu Chen +3
NorMuon: Making Muon more efficient and scalable
Zichong Li, Liming Liu, Chen Liang +2
XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge
Xiaoze Jiang, Yaobo Liang, Weizhu Chen +1
Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
Yueqin Yin, Zhendong Wang, Yujia Xie +2
SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
Xiao Liang, Zhong-Zhi Li, Yeyun Gong +5
Code Execution with Pre-trained Language Models
Chenxiao Liu, Shuai Lu, Weizhu Chen +5
Multi-Task Deep Neural Networks for Natural Language Understanding
Xiaodong Liu, Pengcheng He, Weizhu Chen +1
GLGE: A New General Language Generation Evaluation Benchmark
Dayiheng Liu, Yu Yan, Yeyun Gong +15
Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability
Xiao Liang, Zhong-Zhi Li, Zhenghao Lin +7
Sparse Backpropagation for MoE Training
Liyuan Liu, Jianfeng Gao, Weizhu Chen
Joint Generator-Ranker Learning for Natural Language Generation
Weizhou Shen, Yeyun Gong, Yelong Shen +4
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Qian Liu, Bei Chen, Jiaqi Guo +4
LongRoPE2: Near-Lossless LLM Context Window Scaling
Ning Shang, Li Lyna Zhang, Siyuan Wang +5
Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model
Yueqin Yin, Shentao Yang, Yujia Xie +5
Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR
Xiao Liang, Zhongzhi Li, Yeyun Gong +4
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Kun Zhou, Yeyun Gong, Xiao Liu +8
Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math
Haoran Xu, Baolin Peng, Hany Awadalla +11
CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search
Xiaonan Li, Yeyun Gong, Yelong Shen +7
InfoAgent: Advancing Autonomous Information-Seeking Agents
Gongrui Zhang, Jialiang Zhu, Ruiqi Yang +15
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Greg Yang, Edward J. Hu, Igor Babuschkin +7
Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise
Zhenghao Lin, Yeyun Gong, Yelong Shen +5
FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension
Hsin-Yuan Huang, Chenguang Zhu, Yelong Shen +1
IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles
Tianze Shi, Kedar Tatwawadi, Kaushik Chakrabarti +3
Generation-Augmented Query Expansion For Code Retrieval
Dong Li, Yelong Shen, Ruoming Jin +3
GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation
Biyang Guo, Yeyun Gong, Yelong Shen +4
Multi-LoRA Composition for Image Generation
Ming Zhong, Yelong Shen, Shuohang Wang +6
Finetuning Pretrained Transformers into RNNs
Jungo Kasai, Hao Peng, Yizhe Zhang +6
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
Xiaodong Liu, Pengcheng He, Weizhu Chen +1
Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
Zhendong Wang, Yifan Jiang, Huangjie Zheng +5
ALLSH: Active Learning Guided by Local Sensitivity and Hardness
Shujian Zhang, Chengyue Gong, Xingchao Liu +3
Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
Baolin Peng, Michel Galley, Pengcheng He +8
Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
Zichong Li, Chen Liang, Liliang Ren +3
Diffusion-GAN: Training GANs with Diffusion
Zhendong Wang, Huangjie Zheng, Pengcheng He +2
Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective
Ming Zhong, Chenxin An, Weizhu Chen +2
RE-TRAC: REcursive TRAjectory Compression for Deep Search Agents
Jialiang Zhu, Gongrui Zhang, Xiaolong Ma +17
Adversarial Retriever-Ranker for dense text retrieval
Hang Zhang, Yeyun Gong, Yelong Shen +3
SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
Zichong Li, Chen Liang, Zixuan Zhang +4
The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
Xiaodong Liu, Yu Wang, Jianshu Ji +8
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Qingru Zhang, Minshuo Chen, Alexander Bukharin +5
RLBR: Reinforcement Learning with Biasing Rewards for Contextual Speech Large Language Models
Bo Ren, Ruchao Fan, Yelong Shen +2
Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial Auto-Encoders
Huangjie Zheng, Pengcheng He, Weizhu Chen +1
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Zhengbao Jiang, Yi Mao, Pengcheng He +2
Reasoning Like Program Executors
Xinyu Pi, Qian Liu, Bei Chen +6