The Stan Math Library: Reverse-Mode Automatic Differentiation in C++

Abstract

As computational challenges in optimization and statistical inference grow ever harder, algorithms that utilize derivatives are becoming increasingly more important. The implementation of the derivatives that make these algorithms so powerful, however, is a substantial user burden and the practicality of these algorithms depends critically on tools like automatic differentiation that remove the implementation burden entirely. The Stan Math Library is a C++, reverse-mode automatic differentiation library designed to be usable, extensive and extensible, efficient, scalable, stable, portable, and redistributable in order to facilitate the construction and utilization of such algorithms. Usability is achieved through a simple direct interface and a cleanly abstracted functional interface. The extensive built-in library includes functions for matrix operations, linear algebra, differential equation solving, and most common probability functions. Extensibility derives from a straightforward object-oriented framework for expressions, allowing users to easily create custom functions. Efficiency is achieved through a combination of custom memory management, subexpression caching, traits-based metaprogramming, and expression templates. Partial derivatives for compound functions are evaluated lazily for improved scalability. Stability is achieved by taking care with arithmetic precision in algebraic expressions and providing stable, compound functions where possible. For portability, the library is standards-compliant C++ (03) and has been tested for all major compilers for Windows, Mac OS X, and Linux.

96 pages, 9 figures

Cited by in corpus (23)

• Neural Ordinary Differential Equations
• A Review of automatic differentiation and its efficient implementation
• TensorFlow Distributions
• Automatic Differentiation Variational Inference
• Covariances, Robustness, and Variational Bayes
• Discontinuous Hamiltonian Monte Carlo for discrete parameters and discontinuous likelihoods
• Automatic Differentiation of Rigid Body Dynamics for Optimal Control and Estimation
• Interactive Differentiable Simulation
• MCMC for Bayesian uncertainty quantification from time-series data
• Inverse Transport Networks
• Pathfinder: Parallel quasi-Newton variational inference
• An accelerated linear method for optimizing non-linear wavefunctions in variational Monte Carlo
• The Variational Gaussian Process
• Higher Order Automatic Differentiation of Higher Order Functions
• Hamiltonian Monte Carlo using an adjoint-differentiated Laplace approximation: Bayesian inference for latent Gaussian models and beyond
• Denotational Correctness of Forward-Mode Automatic Differentiation for Iteration and Recursion
• How to Differentiate Collective Variables in Free Energy Codes: Computer-Algebra Code Generation and Automatic Differentiation
• Fast robustness quantification with variational Bayes
• Reality-check for Econophysics: Likelihood-based fitting of physics-inspired market models to empirical data
• Hamiltonian Monte Carlo for Probabilistic Programs with Discontinuities
• Static Analysis for Probabilistic Programs
• A Temporal Kernel Approach for Deep Learning with Continuous-time Information
• A sparse grid approach to balance sheet risk measurement