A companion for the Kiefer--Wolfowitz--Blum stochastic approximation algorithm

Abstract

A stochastic algorithm for the recursive approximation of the location $\theta$ of a maximum of a regression function was introduced by Kiefer and Wolfowitz [Ann. Math. Statist. 23 (1952) 462--466] in the univariate framework, and by Blum [Ann. Math. Statist. 25 (1954) 737--744] in the multivariate case. The aim of this paper is to provide a companion algorithm to the Kiefer--Wolfowitz--Blum algorithm, which allows one to simultaneously recursively approximate the size $\mu$ of the maximum of the regression function. A precise study of the joint weak convergence rate of both algorithms is given; it turns out that, unlike the location of the maximum, the size of the maximum can be approximated by an algorithm which converges at the parametric rate. Moreover, averaging leads to an asymptotically efficient algorithm for the approximation of the couple $(\theta ,\mu )$.

Published in at http://dx.doi.org/10.1214/009053606000001451 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Cited by in corpus (13)

• Convergence rate and averaging of nonlinear two-time-scale stochastic approximation algorithms
• Revisiting Révész's stochastic approximation method for the estimation of a regression function
• Exploiting Higher Order Smoothness in Derivative-free Optimization and Continuous Bandits
• A Robbins-Monro procedure for estimation in semiparametric regression models
• Large and moderate deviation principles for recursive kernel density estimators defined by stochastic approximation method
• Large and moderate deviation principles for averaged stochastic approximation method for the estimation of a regression function
• Optimal two-stage procedures for estimating location and size of the maximum of a multivariate regression function
• Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methods
• Revisiting consistency of a recursive estimator of mixing distributions
• Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm
• Estimation and inference for minimizer and minimum of convex functions: optimality, adaptivity and uncertainty principles
• The stochastic approximation method for the estimation of a multivariate probability density
• Joint behaviour of semirecursive kernel estimators of the location and of the size of the mode of a probability density function