A Geometric Theory of Higher-Order Automatic Differentiation

Abstract

First-order automatic differentiation is a ubiquitous tool across statistics, machine learning, and computer science. Higher-order implementations of automatic differentiation, however, have yet to realize the same utility. In this paper I derive a comprehensive, differential geometric treatment of automatic differentiation that naturally identifies the higher-order differential operators amenable to automatic differentiation as well as explicit procedures that provide a scaffolding for high-performance implementations.

arXiv comment55 pages, 10 figures

References in corpus (3)

• TMB: Automatic Differentiation and Laplace Approximation
• A Conceptual Introduction to Hamiltonian Monte Carlo
• A Review of automatic differentiation and its efficient implementation

Cited by in corpus (6)

• DeepXDE: A deep learning library for solving differential equations
• A Review of automatic differentiation and its efficient implementation
• A Differential-form Pullback Programming Language for Higher-order Reverse-mode Automatic Differentiation
• Higher Order Automatic Differentiation of Higher Order Functions
• Hamiltonian Monte Carlo using an adjoint-differentiated Laplace approximation: Bayesian inference for latent Gaussian models and beyond
• Differentiable Visual Computing