Forward-Mode Automatic Differentiation in Julia
arXiv:1607.07892
Abstract
We present ForwardDiff, a Julia package for forward-mode automatic differentiation (AD) featuring performance competitive with low-level languages like C++. Unlike recently developed AD tools in other popular high-level languages such as Python and MATLAB, ForwardDiff takes advantage of just-in-time (JIT) compilation to transparently recompile AD-unaware user code, enabling efficient support for higher-order differentiation and differentiation using custom number types (including complex numbers). For gradient and Jacobian calculations, ForwardDiff provides a variant of vector-forward mode that avoids expensive heap allocation and makes better use of memory bandwidth than traditional vector mode. In our numerical experiments, we demonstrate that for nontrivially large dimensions, ForwardDiff's gradient computations can be faster than a reverse-mode implementation from the Python-based autograd package. We also illustrate how ForwardDiff is used effectively within JuMP, a modeling language for optimization. According to our usage statistics, 41 unique repositories on GitHub depend on ForwardDiff, with users from diverse fields such as astronomy, optimization, finite element analysis, and statistics. This document is an extended abstract that has been accepted for presentation at the AD2016 7th International Conference on Algorithmic Differentiation.
4 pages
Cited by in corpus (38)
- Fashionable Modelling with Flux
- The software design of Gridap: a Finite Element package based on the Julia JIT compiler
- DiffEqFlux.jl - A Julia Library for Neural Differential Equations
- Adaptive numerical simulations with Trixi.jl: A case study of Julia for scientific computing
- Tropical Tensor Network for Ground States of Spin Glasses
- Safe Control Synthesis via Input Constrained Control Barrier Functions
- Competitive Gradient Descent
- Differentiable Strong Lensing: Uniting Gravity and Neural Nets through Differentiable Probabilistic Programming
- Trajectory optimization for contact-rich motions using implicit differential dynamic programming
- Data-Driven Multi-Grid Solver for Accelerated Pressure Projection
- Inverse Modeling of Viscoelasticity Materials using Physics Constrained Learning
- TensorDiffEq: Scalable Multi-GPU Forward and Inverse Solvers for Physics Informed Neural Networks
- Inference of neutrino flavor evolution through data assimilation and neural differential equations
- Learning battery model parameter dynamics from data with recursive Gaussian process regression
- Diffusion bridges for stochastic Hamiltonian systems and shape evolutions
- On Correctness of Automatic Differentiation for Non-Differentiable Functions
- The Efficiency of Geometric Samplers for Exoplanet Transit Timing Variation Models
- Automatic differentiation for coupled cluster methods
- Arrhenius.jl: A Differentiable Combustion SimulationPackage
- Bayesian modelling of uncertainties of Monte Carlo radiative-transfer simulations
- iLQGames.jl: Rapidly Designing and Solving Differential Games in Julia
- ChemNODE: A Neural Ordinary Differential Equations Approach for Chemical Kinetics Solvers
- Axion-Photon Conversion in Neutron Star Magnetospheres: The Role of the Plasma in the Goldreich-Julian Model
- Cataloging the Visible Universe through Bayesian Inference at Petascale
- Efficient computation of Jacobian matrices for entropy stable summation-by-parts schemes
- Differentiate Everything with a Reversible Embeded Domain-Specific Language
- Worrisome Properties of Neural Network Controllers and Their Symbolic Representations
- Simulation and estimation of a point-process market-model with a matching engine
- Direct optimization of BPX preconditioners
- AbstractDifferentiation.jl: Backend-Agnostic Differentiable Programming in Julia
- Reproducibility Challenge NeurIPS 2019 Report on "Competitive Gradient Descent"
- Estimating the number of entities with vacancies using administrative and online data
- Neural Differential Equations for Inverse Modeling in Model Combustors
- Locally Feasibly Projected Sequential Quadratic Programming for Nonlinear Programming on Arbitrary Smooth Constraint Manifolds
- Single-Look Multi-Master SAR Tomography: An Introduction
- Trust Region Method for Coupled Systems of PDE Solvers and Deep Neural Networks
- Topology Optimization through Differentiable Finite Element Solver
- Flexible numerical optimization with ensmallen