A Review of automatic differentiation and its efficient implementation

Abstract

Derivatives play a critical role in computational statistics, examples being Bayesian inference using Hamiltonian Monte Carlo sampling and the training of neural networks. Automatic differentiation is a powerful tool to automate the calculation of derivatives and is preferable to more traditional methods, especially when differentiating complex algorithms and mathematical functions. The implementation of automatic differentiation however requires some care to insure efficiency. Modern differentiation packages deploy a broad range of computational techniques to improve applicability, run time, and memory management. Among these techniques are operation overloading, region based memory, and expression templates. There also exist several mathematical techniques which can yield high performance gains when applied to complex algorithms. For example, semi-analytical derivatives can reduce by orders of magnitude the runtime required to numerically solve and differentiate an algebraic equation. Open problems include the extension of current packages to provide more specialized routines, and efficient methods to perform higher-order differentiation.

arXiv comment32 pages, 5 figures, submitted for publication. WIREs Data Mining Knowl Discov, March 2019

References in corpus (6)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo
• Automatic Differentiation Variational Inference
• The Stan Math Library: Reverse-Mode Automatic Differentiation in C++
• A General Metric for Riemannian Manifold Hamiltonian Monte Carlo
• A Geometric Theory of Higher-Order Automatic Differentiation

Cited by in corpus (60)

• DeepXDE: A deep learning library for solving differential equations
• Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
• PINNeik: Eikonal solution using physics-informed neural networks
• Forecasting the detection capabilities of third-generation gravitational-wave detectors using $\text{GWFAST}$
• The Science of the Einstein Telescope
• Solving forward and inverse problems of the logarithmic nonlinear Schrodinger equation with PT-symmetric harmonic potential via deep learning
• Adorym: A multi-platform generic x-ray image reconstruction framework based on automatic differentiation
• gradSim: Differentiable simulation for system identification and visuomotor control
• Exponential data encoding for quantum supervised learning
• Quantum Optimal Control via Semi-Automatic Differentiation
• Automatic differentiation for solid mechanics
• Data driven soliton solution of the nonlinear Schrödinger equation with certain $\mathcal{P}\mathcal{T}$-symmetric potentials via deep learning
• Automatic Differentiable Monte Carlo: Theory and Application
• Efficient calculation of gradients in classical simulations of variational quantum algorithms
• Differentiable solver for time-dependent deformation problems with contact
• A Deep Double Ritz Method (D${}^2$RM) for solving Partial Differential Equations using Neural Networks
• Automatic Differentiation for Orbital-Free Density Functional Theory
• SymForce: Symbolic Computation and Code Generation for Robotics
• Mitigating the fermion sign problem by automatic differentiation
• Data-driven rheological characterization of stress buildup and relaxation in thermal greases
• Longitudinal modeling of age-dependent latent traits with generalized additive latent and mixed models
• A Geometric Theory of Higher-Order Automatic Differentiation
• Neural force functional for non-equilibrium many-body colloidal systems
• Bayesian Uncertainty Estimation Under Complex Sampling
• Neural Dynamical Operator: Continuous Spatial-Temporal Model with Gradient-Based and Derivative-Free Optimization Methods
• Differentiable self-consistent space-charge simulation for accelerator design
• The impact of the Universe's expansion rate on constraints on modified growth of structure
• Optimizing Variational Physics-Informed Neural Networks Using Least Squares
• Smoothing Methods for Automatic Differentiation Across Conditional Branches
• dPV: An End-to-End Differentiable Solar-Cell Simulator
• Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
• Relative energies without electronic perturbations via Alchemical Integral Transform
• Automatic differentiation and the optimization of differential equation models in biology
• Inverse-designed dispersive time-varying nanostructures
• Variational Tensor Neural Networks for Deep Learning
• The Inefficiency of Genetic Programming for Symbolic Regression
• Differentiable Optics with dLux I: Deep calibration of Flat Field and Phase Retrieval with Automatic Differentiation
• Automatic differentiation is no panacea for phylogenetic gradient computation
• Optimizing differential equations to fit data and predict outcomes
• The Role of Bases in Quantum Optimal Control
• Numerical simulation of transient heat conduction with moving heat source using Physics Informed Neural Networks
• Efficient CHAD
• ACORNS: An Easy-To-Use Code Generator for Gradients and Hessians
• Differentiable model-based adaptive optics with transmitted and reflected light
• Parametrized multiqubit gate design for neutral-atom based quantum platforms
• A differentiable binary microlensing model using adaptive contour integration method
• An importance sampling approach for reliable and efficient inference in Bayesian ordinary differential equation models
• Enhancing gravitational-wave host localization with SKYFAST: rapid volume and inclination angle reconstruction
• relentless: Transparent, reproducible molecular dynamics simulations for optimization
• Automated Physics-Derived Code Generation for Sensor Fusion and State Estimation
• Private Tabular Survey Data Products through Synthetic Microdata Generation
• Single-layer framework of variational tensor network states
• Unified boson sampling
• Gradient and Hessian approximations in Derivative Free Optimization
• Design of a specimen to train path-dependent deep learning material models from a single uniaxial test: eliciting strain diversity via automatically differentiable elastoplastic topology optimization
• Attitude Estimation of Falcon 9 Rocket Body Based on Photometric Characteristics
• High-Order Hermite Optimization: Fast and Exact Gradient Computation in Open-Loop Quantum Optimal Control using a Discrete Adjoint Approach
• Coarsening Optimization for Differentiable Programming
• Towards Learning Stochastic Population Models by Gradient Descent
• Neural network distillation of orbital dependent density functional theory