A mathematical model for automatic differentiation in machine learning

Abstract

Automatic differentiation, as implemented today, does not have a simple mathematical model adapted to the needs of modern machine learning. In this work we articulate the relationships between differentiation of programs as implemented in practice and differentiation of nonsmooth functions. To this end we provide a simple class of functions, a nonsmooth calculus, and show how they apply to stochastic approximation methods. We also evidence the issue of artificial critical points created by algorithmic differentiation and show how usual methods avoid these points with probability one.

References in corpus (5)

• Learning with Differentiable Perturbed Optimizers
• Provably Correct Automatic Subdifferentiation for Qualified Programs
• Complexity of Finding Stationary Points of Nonsmooth Nonconvex Functions
• Fast Differentiable Sorting and Ranking
• On Correctness of Automatic Differentiation for Non-Differentiable Functions

Cited by in corpus (6)

• Densities of Almost Surely Terminating Probabilistic Programs are Differentiable Almost Everywhere
• Second-order step-size tuning of SGD for non-convex optimization
• On Correctness of Automatic Differentiation for Non-Differentiable Functions
• On the complexity of nonsmooth automatic differentiation
• Stochastic Subgradient Descent Escapes Active Strict Saddles on Weakly Convex Functions
• Conservative parametric optimality and the ridge method for tame min-max problems