Forward-Mode Automatic Differentiation in Julia

Abstract

We present ForwardDiff, a Julia package for forward-mode automatic differentiation (AD) featuring performance competitive with low-level languages like C++. Unlike recently developed AD tools in other popular high-level languages such as Python and MATLAB, ForwardDiff takes advantage of just-in-time (JIT) compilation to transparently recompile AD-unaware user code, enabling efficient support for higher-order differentiation and differentiation using custom number types (including complex numbers). For gradient and Jacobian calculations, ForwardDiff provides a variant of vector-forward mode that avoids expensive heap allocation and makes better use of memory bandwidth than traditional vector mode. In our numerical experiments, we demonstrate that for nontrivially large dimensions, ForwardDiff's gradient computations can be faster than a reverse-mode implementation from the Python-based autograd package. We also illustrate how ForwardDiff is used effectively within JuMP, a modeling language for optimization. According to our usage statistics, 41 unique repositories on GitHub depend on ForwardDiff, with users from diverse fields such as astronomy, optimization, finite element analysis, and statistics. This document is an extended abstract that has been accepted for presentation at the AD2016 7th International Conference on Algorithmic Differentiation.

4 pages

Cited by in corpus (14)

• DiffEqFlux.jl - A Julia Library for Neural Differential Equations
• Inverse Modeling of Viscoelasticity Materials using Physics Constrained Learning
• TensorDiffEq: Scalable Multi-GPU Forward and Inverse Solvers for Physics Informed Neural Networks
• Automatic differentiation for coupled cluster methods
• Arrhenius.jl: A Differentiable Combustion SimulationPackage
• Cataloging the Visible Universe through Bayesian Inference at Petascale
• Worrisome Properties of Neural Network Controllers and Their Symbolic Representations
• Locally Feasibly Projected Sequential Quadratic Programming for Nonlinear Programming on Arbitrary Smooth Constraint Manifolds
• Neural Differential Equations for Inverse Modeling in Model Combustors
• Single-Look Multi-Master SAR Tomography: An Introduction
• Estimating the number of entities with vacancies using administrative and online data
• Trust Region Method for Coupled Systems of PDE Solvers and Deep Neural Networks
• Direct optimization of BPX preconditioners
• Topology Optimization through Differentiable Finite Element Solver