Publications (59)
Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
Canwen Xu, Wangchunshu Zhou, Tao Ge +3
Cant is important for understanding advertising, comedies and dog-whistle politics. However, computational research on cant is hindered by a lack of available datasets. In this pap…
Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study
Tao Ge, Furu Wei, Ming Zhou
Neural sequence-to-sequence (seq2seq) approaches have proven to be successful in grammatical error correction (GEC). Based on the seq2seq framework, we propose a novel fluency boos…
Parallel Data Augmentation for Formality Style Transfer
Yi Zhang, Tao Ge, Xu Sun
The main barrier to progress in the task of Formality Style Transfer is the inadequacy of training data. In this paper, we study how to augment parallel data and propose novel and…
A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model
Xin Sun, Tao Ge, Shuming Ma +3
Synthetic data construction of Grammatical Error Correction (GEC) for non-English languages relies heavily on human-designed and language-specific rules, which produce limited erro…
Pay Attention to Your Tone: Introducing a New Dataset for Polite Language Rewrite
Xun Wang, Tao Ge, Allen Mao +3
We introduce \textsc{PoliteRewrite} -- a dataset for polite language rewrite which is a novel sentence rewrite task. Compared with previous text style transfer tasks that can be mo…
Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Wangchunshu Zhou, Tao Ge, Canwen Xu +2
In this paper, we generalize text infilling (e.g., masked language models) by proposing Sequence Span Rewriting (SSR) as a self-supervised sequence-to-sequence (seq2seq) pre-traini…
Scaling Storm-Resolving Atmospheric AI Simulation to the Entire Planet
Zeyuan Hu, Akshay Subramaniam, Noel Keen +9
Kilometer-scale convection shapes precipitation extremes, tropical organization, and cloud feedbacks, but most global atmospheric models approximate these processes at 25-100 km re…
xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
Xin Cheng, Xun Wang, Xingxing Zhang +5
This paper introduces xRAG, an innovative context compression method tailored for retrieval-augmented generation. xRAG reinterprets document embeddings in dense retrieval--traditio…
Inference with Reference: Lossless Acceleration of Large Language Models
Nan Yang, Tao Ge, Liang Wang +5
We propose LLMA, an LLM accelerator to losslessly speed up Large Language Model (LLM) inference with references. LLMA is motivated by the observation that there are abundant identi…
Enhancing Language Model Rationality with Bi-Directional Deliberation Reasoning
Yadong Zhang, Shaoguang Mao, Wenshan Wu +4
This paper introduces BI-Directional DEliberation Reasoning (BIDDER), a novel reasoning approach to enhance the decision rationality of language models. Traditional reasoning metho…
Exploiting Task-Oriented Resources to Learn Word Embeddings for Clinical Abbreviation Expansion
Yue Liu, Tao Ge, Kusum S. Mathews +2
In the medical domain, identifying and expanding abbreviations in clinical texts is a vital task for both better human and machine understanding. It is a challenging task because m…
SCALE: Synergized Collaboration of Asymmetric Language Translation Engines
Xin Cheng, Xun Wang, Tao Ge +4
In this paper, we introduce SCALE, a collaborative framework that connects compact Specialized Translation Models (STMs) and general-purpose Large Language Models (LLMs) as one uni…
Scheduled DropHead: A Regularization Method for Transformer Models
Wangchunshu Zhou, Tao Ge, Ke Xu +2
In this paper, we introduce DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of transformer…
Lossless Acceleration for Seq2seq Generation with Aggressive Decoding
Tao Ge, Heming Xia, Xin Sun +2
We study lossless acceleration for seq2seq generation with a novel decoding algorithm -- Aggressive Decoding. Unlike the previous efforts (e.g., non-autoregressive decoding) speedi…
Extensible Prompts for Language Models on Zero-shot Language Style Customization
Tao Ge, Jing Hu, Li Dong +5
We propose eXtensible Prompt (X-Prompt) for prompting a large language model (LLM) beyond natural language (NL). X-Prompt instructs an LLM with not only NL but also an extensible v…
Residual Corrective Diffusion Modeling for Km-scale Atmospheric Downscaling
Morteza Mardani, Noah Brenowitz, Yair Cohen +10
The state of the art for physical hazard prediction from weather and climate requires expensive km-scale numerical simulations driven by coarser resolution global inputs. Here, a g…
Formality Style Transfer with Hybrid Textual Annotations
Ruochen Xu, Tao Ge, Furu Wei
Formality style transformation is the task of modifying the formality of a given sentence without changing its content. Its challenge is the lack of large-scale sentence-aligned pa…
DL-Corrector-Remapper: A grid-free bias-correction deep learning methodology for data-driven high-resolution global weather forecasting
Tao Ge, Jaideep Pathak, Akshay Subramaniam +1
Data-driven models, such as FourCastNet (FCN), have shown exemplary performance in high-resolution global weather forecasting. This performance, however, is based on supervision on…
K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoning
Yadong Zhang, Shaoguang Mao, Tao Ge +4
Strategic reasoning is a complex yet essential capability for intelligent agents. It requires Large Language Model (LLM) agents to adapt their strategies dynamically in multi-agent…
LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
Yadong Zhang, Shaoguang Mao, Tao Ge +7
This paper presents a comprehensive survey of the current status and opportunities for Large Language Models (LLMs) in strategic reasoning, a sophisticated form of reasoning that n…
Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
Xin Sun, Tao Ge, Furu Wei +1
In this paper, we propose Shallow Aggressive Decoding (SAD) to improve the online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC). SAD…
Scaling Synthetic Data Creation with 1,000,000,000 Personas
Tao Ge, Xin Chan, Xiaoyang Wang +3
We propose a novel persona-driven data synthesis methodology that leverages various perspectives within a large language model (LLM) to create diverse synthetic data. To fully expl…
Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers
Shwai He, Tao Ge, Guoheng Sun +3
Traditional transformer models often allocate a fixed amount of computational resources to every input token, leading to inefficient and unnecessary computation. To address this, t…
Improving Grammatical Error Correction with Machine Translation Pairs
Wangchunshu Zhou, Tao Ge, Chang Mu +3
We propose a novel data synthesis method to generate diverse error-corrected sentence pairs for improving grammatical error correction, which is based on a pair of machine translat…
A Metal Artifact Reduction Scheme For Accurate Iterative Dual-Energy CT Algorithms
Tao Ge, Maria Medrano, Rui Liao +4
CT images have been used to generate radiation therapy treatment plans for more than two decades. Dual-energy CT (DECT) has shown high accuracy in estimating electronic density or…
LEDGER: Scaling Agentic Document Editing with Dependency-aware Graph Retrieval
Mike Hang Wang, Utkarsh Garg, Reza Davari +5
We introduce LEDGER to tackle the novel context engineering challenge of agentic document editing, where localized edits to long, structured documents must be applied efficiently w…
In-context Autoencoder for Context Compression in a Large Language Model
Tao Ge, Jing Hu, Lei Wang +3
We propose the In-context Autoencoder (ICAE), leveraging the power of a large language model (LLM) to compress a long context into short compact memory slots that can be directly c…
TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
Leitian Tao, Baolin Peng, Wenlin Yao +5
The paper proposes TRACE, a turn-level reward assignment method that estimates credit for each tool-call in long-horizon agents using log‑ratio state values and temporal‑difference…
Aligning Coordinated Text Streams through Burst Information Network Construction and Decipherment
Tao Ge, Qing Dou, Xiaoman Pan +5
Aligning coordinated text streams from multiple sources and multiple languages has opened many new research venues on cross-lingual knowledge discovery. In this paper we aim to adv…
Low-code LLM: Graphical User Interface over Large Language Models
Yuzhe Cai, Shaoguang Mao, Wenshan Wu +10
Utilizing Large Language Models (LLMs) for complex tasks is challenging, often involving a time-consuming and uncontrollable prompt engineering process. This paper introduces a nov…
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
Heming Xia, Zhe Yang, Qingxiu Dong +6
To mitigate the high inference latency stemming from autoregressive decoding in Large Language Models (LLMs), Speculative Decoding has emerged as a novel decoding paradigm for LLM…
ParallelSpec: Parallel Drafter for Efficient Speculative Decoding
Zilin Xiao, Hongming Zhang, Tao Ge +3
Speculative decoding has proven to be an efficient solution to large language model (LLM) inference, where the small drafter predicts future tokens at a low cost, and the target mo…
ALYMPICS: LLM Agents Meet Game Theory -- Exploring Strategic Decision-Making with AI Agents
Shaoguang Mao, Yuzhe Cai, Yan Xia +5
This paper introduces Alympics (Olympics for Agents), a systematic simulation framework utilizing Large Language Model (LLM) agents for game theory research. Alympics creates a ver…
Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction
Mengyun Chen, Tao Ge, Xingxing Zhang +2
We propose a novel language-independent approach to improve the efficiency for Grammatical Error Correction (GEC) by dividing the task into two subtasks: Erroneous Span Detection (…
Orchard: An Open-Source Agentic Modeling Framework
Baolin Peng, Wenlin Yao, Qianhui Wu +11
Agentic modeling aims to transform LLMs into autonomous agents capable of solving complex tasks through planning, reasoning, tool use, and multi-turn interaction with external envi…
Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
Yuheng Zhang, Dian Yu, Tao Ge +5
Reinforcement learning from human feedback (RLHF) has demonstrated remarkable effectiveness in aligning large language models (LLMs) with human preferences. Many existing alignment…
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
Heming Xia, Tao Ge, Peiyi Wang +3
We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Specu…
Fact-aware Sentence Split and Rephrase with Permutation Invariant Training
Yinuo Guo, Tao Ge, Furu Wei
Sentence Split and Rephrase aims to break down a complex sentence into several simple sentences with its meaning preserved. Previous studies tend to address the issue by seq2seq le…
Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory
Guangyue Peng, Tao Ge, Wen Luo +4
Semiparametric language models (LMs) have shown promise in various Natural Language Processing (NLP) tasks. However, they utilize non-parametric memory as static storage, which lac…
Synthetic Computers at Scale for Long-Horizon Productivity Simulation
Tao Ge, Baolin Peng, Hao Cheng +1
Realistic long-horizon productivity work is strongly conditioned on user-specific computer environments, where much of the work context is stored and organized through directory st…
A Machine-learning Based Initialization for Joint Statistical Iterative Dual-energy CT with Application to Proton Therapy
Tao Ge, Maria Medrano, Rui Liao +3
Dual-energy CT (DECT) has been widely investigated to generate more informative and more accurate images in the past decades. For example, Dual-Energy Alternating Minimization (DEA…
Climate in a Bottle: Towards a Generative Foundation Model for the Kilometer-Scale Global Atmosphere
Noah D. Brenowitz, Tao Ge, Akshay Subramaniam +7
Climate modeling is reaching unprecedented resolution, producing petabytes of data. AI climate model emulators offer a path to computationally cheap analysis, enabling new scientif…
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Canwen Xu, Wangchunshu Zhou, Tao Ge +3
Recent studies on compression of pretrained language models (e.g., BERT) usually use preserved accuracy as the metric for evaluation. In this paper, we propose two new metrics, lab…
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
Canwen Xu, Wangchunshu Zhou, Tao Ge +2
In this paper, we propose a novel model compression approach to effectively compress BERT by progressive module replacing. Our approach first divides the original BERT into several…
BERT Loses Patience: Fast and Robust Inference with Early Exit
Wangchunshu Zhou, Canwen Xu, Tao Ge +3
In this paper, we propose Patience-based Early Exit, a straightforward yet effective inference method that can be used as a plug-and-play technique to simultaneously improve the ef…
Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu +3
Human intelligence thrives on cognitive synergy, where collaboration among different minds yield superior outcomes compared to isolated individuals. In this work, we propose Solo P…
Smart Word Suggestions for Writing Assistance
Chenshuo Wang, Shaoguang Mao, Tao Ge +5
Enhancing word usage is a desired feature for writing assistance. To further advance research in this area, this paper introduces "Smart Word Suggestions" (SWS) task and benchmark.…
EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation
Tao Ge, Si-Qing Chen, Furu Wei
We introduce EdgeFormer -- a parameter-efficient Transformer for on-device seq2seq generation under the strict computation and memory constraints. Compared with the previous parame…
MB-DECTNet: A Model-Based Unrolled Network for Accurate 3D DECT Reconstruction
Tao Ge, Maria Medrano, Rui Liao +4
Numerous dual-energy CT (DECT) techniques have been developed in the past few decades. Dual-energy CT (DECT) statistical iterative reconstruction (SIR) has demonstrated its potenti…
Pseudo-Bidirectional Decoding for Local Sequence Transduction
Wangchunshu Zhou, Tao Ge, Ke Xu
Local sequence transduction (LST) tasks are sequence transduction tasks where there exists massive overlapping between the source and target sequences, such as Grammatical Error Co…
Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting
Yi Zhang, Tao Ge, Furu Wei +2
We study sequence-to-sequence (seq2seq) pre-training with data augmentation for sentence rewriting. Instead of training a seq2seq model with gold training data and augmented data s…
OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personas
Xiaoyang Wang, Hongming Zhang, Tao Ge +3
Customizable role-playing in large language models (LLMs), also known as character generalization, is gaining increasing attention for its versatility and cost-efficiency in develo…
Text Revision by On-the-Fly Representation Optimization
Jingjing Li, Zichao Li, Tao Ge +2
Text revision refers to a family of natural language generation tasks, where the source and target sequences share moderate resemblance in surface form but differentiate in attribu…
Learn Beyond The Answer: Training Language Models with Reflection for Mathematical Reasoning
Zhihan Zhang, Tao Ge, Zhenwen Liang +5
Supervised fine-tuning enhances the problem-solving abilities of language models across various mathematical reasoning tasks. To maximize such benefits, existing research focuses o…
Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens
Xu Ouyang, Tao Ge, Thomas Hartvigsen +3
We reveal that low-bit quantization favors undertrained large language models (LLMs) by observing that models with larger sizes or fewer training tokens experience less quantizatio…
Proximal Newton Methods for X-Ray Imaging with Non-Smooth Regularization
Tao Ge, Umberto Villa, Ulugbek S. Kamilov +1
Non-smooth regularization is widely used in image reconstruction to eliminate the noise while preserving subtle image structures. In this work, we investigate the use of proximal N…
Self-Adversarial Learning with Comparative Discrimination for Text Generation
Wangchunshu Zhou, Tao Ge, Ke Xu +2
Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated…
Refining Corpora from a Model Calibration Perspective for Chinese Spelling Correction
Dingyao Yu, Yang An, Wei Ye +4
Chinese Spelling Correction (CSC) commonly lacks large-scale high-quality corpora, due to the labor-intensive labeling of spelling errors in real-life human writing or typing scena…
DocReward: A Document Reward Model for Structuring and Stylizing
Junpeng Liu, Yuzhong Zhao, Bowen Cao +17
Recent agentic workflows automate professional document generation but focus narrowly on textual quality, overlooking structural and stylistic professionalism, which is equally cri…