Adaptive two-points zero-order accelerated methods for stochastic convex optimization problems

Abstract

We study stochastic convex optimization, where each sampled function can be evaluated at two arbitrary points. Our accelerated method chooses step sizes, batch sizes and smoothing radii from function values. A forward difference between smoothing scales controls the radius; we account for changes in the smoothed objective. Under the stated regularity and second-moment assumptions, exact function evaluations give \begin{multline*} \mathbb{E}\bigl[f(x_N)-f(x^\star)\bigr] \le \widetilde{O}\biggl( \frac{\sqrt{d}\,M_2 D}{\sqrt{T}} +\min\biggl( \frac{L D^2}{N^2},\; \frac{d^{1/4} M D}{N}+\frac{\sqrt{d}\,M D^2}{γ_{\max} N^2} \biggr)\biggr). \end{multline*} Here $N,T,d,D$ denote the numbers of optimization steps and paired evaluations, the dimension and a feasible-set diameter bound, respectively. Evaluations are available within Euclidean distance ${\gamma }_{\max }$ of the feasible set. For the population objective $f(x)={\mathbb{E}}_{\xi }F(x,\xi )$, $M$ satisfies $|f(x)-f(y)|\le M\Vert x-y{\Vert }_2$, and $M_2$ satisfies ${\mathbb{E}}_{\xi }\Vert \nabla F(x,\xi ){\Vert }_2^2\le M_2^2$ for sample weak gradients in the interior of the evaluation domain. The method requires no $L,M,M_2$, where $L$ is a Lipschitz constant of $\nabla f$; set $L=+\infty$ without smoothness. The method uses exactly $2T$ values in $3N$ sequential sampling rounds with predetermined batches, or $5N$ with batches chosen from observations. An extension selects a search radius without a supplied distance bound. Fixed radius parameters can leave nonvanishing error; suitable choices depending on the budgets yield convergence.