5 papers
Gradient-Hessian Query Tradeoffs: Matching Lower Bounds in Logarithmic Dimension
Wendao Wu, Haihan Zhang, Chenheng Zhang +5
We establish matching lower bounds for the tradeoff between gradient and Hessian queries in nonconvex optimization with Lipschitz Hessian. In normalized units, the initial objectiv…
Dimension-Optimal Dynamic Regret in Exp-Concave Online Learning
Yanyi Li, Wendao Wu, Haihan Zhang +6
Tracking changing comparators is a central problem in online learning. For exp-concave losses, previous work established the dependence on the horizon and Eu…
The Minimax Cost of Smoothness in Fully First-Order Stochastic Bilevel Optimization
Wendao Wu, Haihan Zhang, Chenheng Zhang +5
We study the sample complexity of finding a point with expected hypergradient norm at most in nonconvex--strongly-convex stochastic bilevel optimization using fresh, globally…
Closing the Accuracy Gap in Stochastic First-Order Bilevel Optimization
Haihan Zhang, Wendao Wu, Chenheng Zhang +5
We characterize the optimal accuracy dependence for smooth nonconvex--strongly-convex bilevel optimization with stochastic first-order oracles. For globally unbiased fresh-gradient…
PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training
Yanyi Li, Yimu Zhang, Cong Fang
Activations have become the primary memory bottleneck in large-batch LLM training. However, existing compression methods fail to exploit the spectral structure of activations, resu…