112 citations · 144 across the 40 of their papers we have counts for
23 papers · 1 filter
Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
Yi Ding, Ruqi Zhang
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR).…
Slithering Through Gaps: Capturing Discrete Isolated Modes via Logistic Bridging
Pinaki Mohanty, Ruqi Zhang
High-dimensional and complex discrete distributions often exhibit multimodal behavior due to inherent discontinuities, posing significant challenges for sampling. Gradient-based di…
Generative Frontiers: Why Evaluation Matters for Diffusion Language Models
Patrick Pynadath, Jiaxin Shi, Ruqi Zhang
Diffusion language models have seen exciting recent progress, offering far more flexibility in generative trajectories than autoregressive models. This flexibility has motivated a…
CANDI: Hybrid Discrete-Continuous Diffusion Models
Patrick Pynadath, Jiaxin Shi, Ruqi Zhang
While continuous diffusion has shown remarkable success in continuous domains such as image generation, its direct application to discrete data has underperformed pure discrete for…
Stacey: Promoting Stochastic Steepest Descent via Accelerated -Smooth Nonconvex Optimization
Xinyu Luo, Cedar Site Bai, Bolian Li +3
While popular optimization methods such as SGD, AdamW, and Lion depend on steepest descent updates in either or norms, there remains a critical gap in handli…
Accelerating Inference of Discrete Autoregressive Normalizing Flows by Selective Jacobi Decoding
Jiaru Zhang, Juanwu Lu, Xiaoyu Wu +2
Discrete normalizing flows are promising generative models with advantages such as analytical log-likelihood computation and end-to-end training. However, the architectural constra…
Bayesian Computation in Deep Learning
Wenlong Chen, Bolian Li, Ruqi Zhang +1
Bayesian methods have shown success in deep learning applications. For example, in predictive tasks, Bayesian neural networks leverage Bayesian reasoning of model uncertainty to im…
Making Reliable and Flexible Decisions in Long-tailed Classification
Bolian Li, Ruqi Zhang
Long-tailed classification is challenging due to its heavy imbalance in class probabilities. While existing methods often focus on overall accuracy or accuracy for tail classes, th…
Exploring Non-Convex Discrete Energy Landscapes: An Efficient Langevin-Like Sampler with Replica Exchange
Haoyang Zheng, Hengrong Du, Ruqi Zhang +1
Gradient-based Discrete Samplers (GDSs) are effective for sampling discrete energy landscapes. However, they often stagnate in complex, non-convex settings. To improve exploration,…
Training Bayesian Neural Networks with Sparse Subspace Variational Inference
Junbo Li, Zichen Miao, Qiang Qiu +1
Bayesian neural networks (BNNs) offer uncertainty quantification but come with the downside of substantially increased training and inference costs. Sparse BNNs have been investiga…
Gradient-based Discrete Sampling with Automatic Cyclical Scheduling
Patrick Pynadath, Riddhiman Bhattacharya, Arun Hariharan +1
Discrete distributions, particularly in high-dimensional deep models, are often highly multimodal due to inherent discontinuities. While gradient-based discrete sampling has proven…
Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI
Theodore Papamarkou, Maria Skoularidou, Konstantina Palla +22
In the current landscape of deep learning research, there is a predominant emphasis on achieving high predictive accuracy in supervised tasks involving large image and language dat…
Entropy-MCMC: Sampling from Flat Basins with Ease
Bolian Li, Ruqi Zhang
Bayesian deep learning counts on the quality of posterior distribution estimation. However, the posterior of deep neural networks is highly multi-modal in nature, with local modes…
Long-tailed Classification from a Bayesian-decision-theory Perspective
Bolian Li, Ruqi Zhang
Long-tailed classification poses a challenge due to its heavy imbalance in class probabilities and tail-sensitivity risks with asymmetric misprediction costs. Recent attempts have…
DP-Fast MH: Private, Fast, and Accurate Metropolis-Hastings for Large-Scale Bayesian Inference
Wanrong Zhang, Ruqi Zhang
Bayesian inference provides a principled framework for learning from complex data and reasoning under uncertainty. It has been widely applied in machine learning tasks such as medi…
Calibrating the Rigged Lottery: Making All Tickets Reliable
Bowen Lei, Ruqi Zhang, Dongkuan Xu +1
Although sparse training has been successfully used in various resource-limited deep learning tasks to save memory, accelerate training, and reduce inference time, the reliability…
Sampling in Constrained Domains with Orthogonal-Space Variational Gradient Descent
Ruqi Zhang, Qiang Liu, Xin T. Tong
Sampling methods, as important inference and learning techniques, are typically designed for unconstrained domains. However, constraints are ubiquitous in machine learning problems…
A Langevin-like Sampler for Discrete Distributions
Ruqi Zhang, Xingchao Liu, Qiang Liu
We propose discrete Langevin proposal (DLP), a simple and scalable gradient-based proposal for sampling complex high-dimensional discrete distributions. In contrast to Gibbs sampli…
Low-Precision Stochastic Gradient Langevin Dynamics
Ruqi Zhang, Andrew Gordon Wilson, Christopher De Sa
While low-precision optimization has been widely used to accelerate deep learning, low-precision sampling remains largely unexplored. As a consequence, sampling is simply infeasibl…
Asymptotically Optimal Exact Minibatch Metropolis-Hastings
Ruqi Zhang, A. Feder Cooper, Christopher De Sa
Metropolis-Hastings (MH) is a commonly-used MCMC algorithm, but it can be intractable on large datasets due to requiring computations over the whole dataset. In this paper, we stud…
AMAGOLD: Amortized Metropolis Adjustment for Efficient Stochastic Gradient MCMC
Ruqi Zhang, A. Feder Cooper, Christopher De Sa
Stochastic gradient Hamiltonian Monte Carlo (SGHMC) is an efficient method for sampling from continuous distributions. It is a faster alternative to HMC: instead of using the whole…
Poisson-Minibatching for Gibbs Sampling with Convergence Rate Guarantees
Ruqi Zhang, Christopher De Sa
Gibbs sampling is a Markov chain Monte Carlo method that is often used for learning and inference on graphical models. Minibatching, in which a small random subset of the graph is…
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
Ruqi Zhang, Chunyuan Li, Jianyi Zhang +2
The posteriors over neural network weights are high dimensional and multimodal. Each mode typically characterizes a meaningfully different representation of the data. We develop Cy…