25 citations · 98 across the 17 of their papers we have counts for
17 papers
Positional Description Matters for Transformers Arithmetic
Ruoqi Shen, Sébastien Bubeck, Ronen Eldan +3
Transformers, central to the successes in modern Natural Language Processing, often falter on arithmetic tasks despite their vast capabilities --which paradoxically include remarka…
FiLM: Fill-in Language Models for Any-Order Generation
Tianxiao Shen, Hao Peng, Ruoqi Shen +3
Language models have become the backbone of today's AI systems. However, their predominant left-to-right generation limits the use of bidirectional context, which is essential for…
Algorithmic Aspects of the Log-Laplace Transform and a Non-Euclidean Proximal Sampler
Sivakanth Gopi, Yin Tat Lee, Daogao Liu +2
The development of efficient sampling algorithms catering to non-Euclidean geometries has been a challenging endeavor, as discretization techniques which succeed in the Euclidean s…
How to Fine-Tune Vision Models with SGD
Ananya Kumar, Ruoqi Shen, Sebastien Bubeck +1
SGD and AdamW are the two most used optimizers for fine-tuning large neural networks in computer vision. When the two methods perform the same, SGD is preferable because it uses le…
Condition-number-independent convergence rate of Riemannian Hamiltonian Monte Carlo with numerical integrators
Yunbum Kook, Yin Tat Lee, Ruoqi Shen +1
We study the convergence rate of discretized Riemannian Hamiltonian Monte Carlo on sampling from distributions in the form of on a convex body $\mathcal{M}\subset\mathb…
Private Convex Optimization in General Norms
Sivakanth Gopi, Yin Tat Lee, Daogao Liu +2
We propose a new framework for differentially private optimization of convex functions which are Lipschitz in an arbitrary norm . Our algorithms are based on a regulariz…
Data Augmentation as Feature Manipulation
Ruoqi Shen, Sébastien Bubeck, Suriya Gunasekar
Data augmentation is a cornerstone of the machine learning pipeline, yet its theoretical underpinnings remain unclear. Is it merely a way to artificially augment the data set size?…
On Optimal Early Stopping: Over-informative versus Under-informative Parametrization
Ruoqi Shen, Liyao Gao, Yi-An Ma
Early stopping is a simple and widely used method to prevent over-training neural networks. We develop theoretical results to reveal the relationship between the optimal early stop…
Sampling with Riemannian Hamiltonian Monte Carlo in a Constrained Space
Yunbum Kook, Yin Tat Lee, Ruoqi Shen +1
We demonstrate for the first time that ill-conditioned, non-smooth, constrained distributions in very high dimension, upwards of 100,000, can be sampled efficiently $\textit{in pra…
Lower Bounds on Metropolized Sampling Methods for Well-Conditioned Distributions
Yin Tat Lee, Ruoqi Shen, Kevin Tian
We give lower bounds on the performance of two of the most popular sampling methods in practice, the Metropolis-adjusted Langevin algorithm (MALA) and multi-step Hamiltonian Monte…
Near-Optimal Randomized Exploration for Tabular Markov Decision Processes
Zhihan Xiong, Ruoqi Shen, Qiwen Cui +2
We study algorithms using randomized value functions for exploration in reinforcement learning. This type of algorithms enjoys appealing empirical performance. We show that when we…
Structured Logconcave Sampling with a Restricted Gaussian Oracle
Yin Tat Lee, Ruoqi Shen, Kevin Tian
We give algorithms for sampling several structured logconcave families to high accuracy. We further develop a reduction framework, inspired by proximal point methods in convex opti…
Generalized Leverage Score Sampling for Neural Networks
Jason D. Lee, Ruoqi Shen, Zhao Song +2
Leverage score sampling is a powerful technique that originates from theoretical computer science, which can be used to speed up a large number of fundamental questions, e.g. linea…
Composite Logconcave Sampling with a Restricted Gaussian Oracle
Ruoqi Shen, Kevin Tian, Yin Tat Lee
We consider sampling from composite densities on of the form for well-conditioned and convex (but possibly non-smooth) ,…
When is Particle Filtering Efficient for Planning in Partially Observed Linear Dynamical Systems?
Simon S. Du, Wei Hu, Zhiyuan Li +3
Particle filtering is a popular method for inferring latent states in stochastic dynamical systems, whose theoretical properties have been well studied in machine learning and stat…
Logsmooth Gradient Concentration and Tighter Runtimes for Metropolized Hamiltonian Monte Carlo
Yin Tat Lee, Ruoqi Shen, Kevin Tian
We show that the gradient norm for , where is strongly convex and smooth, concentrates tightly around its mean. This removes a barrier in…
The Randomized Midpoint Method for Log-Concave Sampling
Ruoqi Shen, Yin Tat Lee
Sampling from log-concave distributions is a well researched problem that has many applications in statistics and machine learning. We study the distributions of the form $p^{*}\pr…