SAGE: A Set-based Adaptive Gradient Estimator

Abstract

We introduce a set membership perspective on gradient estimation for black-box scalar functions, in which the gradient is not approximated as a single point estimate but characterised as a member of a set of admissible gradients consistent with available function samples and regularity assumptions. Bounds on gradient uncertainty, obtained from standard multivariate Taylor series results, are used to represent this admissible set through linear inequalities, making gradient uncertainty explicit and directly exploitable. Building on this formulation, we propose an adaptive strategy to refine the admissible gradient set to a desired precision by selectively acquiring new samples only when necessary. The resulting framework naturally accommodates function evaluations affected by noise with finite but a priori unknown bounds, yields the theoretical best attainable precision of the gradient set, and provides the optimal sampling distance from the point of interest to maximise information gain during refinement. Within this framework, we introduce the Set-Based Adaptive Gradient Estimator (SAGE), which extracts an estimate of the true gradient from the admissible set whilst preserving high sample efficiency and robustness to noise. Numerical experiments on large collections of random gradient-query points, as well as in the context of a first-order optimisation method, compared the performance of SAGE against commonly used gradient estimators from the literature. Our results show that SAGE achieved competitive gradient accuracy across noise levels without requiring any a priori noise characterisation. In a first-order optimisation setting, noise-informed competitors can outperform SAGE when such characterisation is available. Nevertheless, SAGE remained broadly competitive, making it particularly attractive in settings where noise characterisation is impractical to obtain.