849 citations · 2.4k across the 172 of their papers we have counts for
60 papers · 1 filter
From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing
Guannan Lai, Haoran Hu, Long Chen +2
Existing LLM routing methods often construct supervision from a single sampled response for each query--model pair. Because LLM generation is stochastic, however, such an observati…
Masked Diffusion Models as Energy Minimization
Sitong Chen, Shen Nie, Jiacheng Sun +4
We present a systematic theoretical framework that interprets masked diffusion models (MDMs) as solutions to energy minimization problems in discrete optimal transport. Specificall…
Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
Shuchen Xue, Tianyu Xie, Tianyang Hu +5
Efficiently scaling Large Language Models (LLMs) necessitates exploring alternatives to dominant autoregressive (AR) methods, with Masked Diffusion Models (MDMs) emerging as candid…
Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling
Tianyu Xie, Shuchen Xue, Zijin Feng +4
Discrete diffusion models have recently shown great promise for modeling complex discrete data, with masked diffusion models (MDMs) offering a compelling trade-off between quality…
Implicit Search via Discrete Diffusion: A Study on Chess
Jiacheng Ye, Zhenyu Wu, Jiahui Gao +4
In the post-AlphaGo era, there has been a renewed interest in search techniques such as Monte Carlo Tree Search (MCTS), particularly in their application to Large Language Models (…
Improved Diffusion-based Generative Model with Better Adversarial Robustness
Zekun Wang, Mingyang Yi, Shuchen Xue +4
Diffusion Probabilistic Models (DPMs) have achieved significant success in generative tasks. However, their training and sampling processes suffer from the issue of distribution mi…
How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training
Yixin Ou, Yunzhi Yao, Ningyu Zhang +5
Despite exceptional capabilities in knowledge-intensive tasks, Large Language Models (LLMs) face a critical gap in understanding how they internalize new knowledge, particularly ho…
ProofAug: Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis
Haoxiong Liu, Jiacheng Sun, Zhenguo Li +1
The synergy between deep learning models and traditional automation tools, such as built-in tactics of the proof assistant and off-the-shelf automated theorem provers, plays a cruc…
Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models
Kaican Li, Weiyan Xie, Yongxiang Huang +5
Fine-tuning foundation models often compromises their robustness to distribution shifts. To remedy this, most robust fine-tuning methods aim to preserve the pre-trained features. H…
How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs
Guhao Feng, Kai Yang, Yuntian Gu +6
Despite the remarkable success of Transformer-based large language models (LLMs) across various domains, understanding and enhancing their mathematical capabilities remains a signi…
Jailbreaking as a Reward Misspecification Problem
Zhihui Xie, Jiahui Gao, Lei Li +3
The widespread adoption of large language models (LLMs) has raised concerns about their safety and reliability, particularly regarding their vulnerability to adversarial attacks. I…
QuickLLaMA: Query-aware Inference Acceleration for Large Language Models
Jingyao Li, Han Shi, Xin Jiang +3
The capacity of Large Language Models (LLMs) to comprehend and reason over long contexts is pivotal for advancements in diverse fields. Yet, they still stuggle with capturing long-…
Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data
Jingyang Ou, Shen Nie, Kaiwen Xue +4
Discrete diffusion models with absorbing processes have shown promise in language modeling. The key quantities to be estimated are the ratios between the marginal probabilities of…
The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling
Jiajun Ma, Shuchen Xue, Tianyang Hu +5
With the incorporation of the UNet architecture, diffusion probabilistic models have become a dominant force in image generation tasks. One key design in UNet is the skip connectio…
AlgoFormer: An Efficient Transformer Framework with Algorithmic Structures
Yihang Gao, Chuanyang Zheng, Enze Xie +6
Besides natural language processing, transformers exhibit extraordinary performance in solving broader applications, including scientific computing and computer vision. Previous wo…
Robustness May be More Brittle than We Think under Different Degrees of Distribution Shifts
Kaican Li, Yifan Zhang, Lanqing Hong +2
Out-of-distribution (OOD) generalization is a complicated problem due to the idiosyncrasies of possible distribution shifts between training and test domains. Most benchmarks emplo…
BYOM: Building Your Own Multi-Task Model For Free
Weisen Jiang, Baijiong Lin, Han Shi +3
Recently, various merging methods have been proposed to build a multi-task model from task-specific finetuned models without retraining. However, existing methods suffer from a lar…
Explore and Exploit the Diverse Knowledge in Model Zoo for Domain Generalization
Yimeng Chen, Tianyang Hu, Fengwei Zhou +2
The proliferation of pretrained models, as a result of advancements in pretraining techniques, has led to the emergence of a vast zoo of publicly available models. Effectively util…
On the Generalization of Diffusion Model
Mingyang Yi, Jiacheng Sun, Zhenguo Li
The diffusion probabilistic generative models are widely used to generate high-quality data. Though they can synthetic data that does not exist in the training set, the rationale b…
Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models
Weijian Luo, Tianyang Hu, Shifeng Zhang +3
Due to the ease of training, ability to scale, and high sample quality, diffusion models (DMs) have become the preferred option for generative modeling, with numerous pre-trained m…
Fair-CDA: Continuous and Directional Augmentation for Group Fairness
Rui Sun, Fengwei Zhou, Zhenhua Dong +6
In this work, we propose {\it Fair-CDA}, a fine-grained data augmentation strategy for imposing fairness constraints. We use a feature disentanglement method to extract the feature…
Boosting Out-of-Distribution Detection with Multiple Pre-trained Models
Feng Xue, Zi He, Chuanlong Xie +2
Out-of-Distribution (OOD) detection, i.e., identifying whether an input is sampled from a novel distribution other than the training distribution, is a critical task for safely dep…
ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization
Qishi Dong, Awais Muhammad, Fengwei Zhou +5
Recent advances on large-scale pre-training have shown great potentials of leveraging a large set of Pre-Trained Models (PTMs) for improving Out-of-Distribution (OoD) generalizatio…
Learning to Prove Trigonometric Identities
Zhou Liu, Yujun Li, Zhengying Liu +2
Automatic theorem proving with deep learning methods has attracted attentions recently. In this paper, we construct an automatic proof system for trigonometric identities. We defin…