Differentiate Everything with a Reversible Embeded Domain-Specific Language

Abstract

Reverse-mode automatic differentiation (AD) suffers from the issue of having too much space overhead to trace back intermediate computational states for back-propagation. The traditional method to trace back states is called checkpointing that stores intermediate states into a global stack and restore state through either stack pop or re-computing. The overhead of stack manipulations and re-computing makes the general purposed (not tensor-based) AD engines unable to meet many industrial needs. Instead of checkpointing, we propose to use reverse computing to trace back states by designing and implementing a reversible programming eDSL, where a program can be executed bi-directionally without implicit stack operations. The absence of implicit stack operations makes the program compatible with existing compiler features, including utilizing existing optimization passes and compiling the code as GPU kernels. We implement AD for sparse matrix operations and some machine learning applications to show that our framework has the state-of-the-art performance.

arXiv commentGithub: https://github.com/GiggleLiu/NiLang.jl

References in corpus (28)

• Adam: A Method for Stochastic Optimization
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• NICE: Non-linear Independent Components Estimation
• Normalizing Flows: An Introduction and Review of Current Methods
• Density estimation using Real NVP
• Training Deep Nets with Sublinear Memory Cost
• An improved Landauer Principle with finite-size corrections
• PennyLane: Automatic differentiation of hybrid quantum-classical computations
• Differentiable Programming Tensor Networks
• Unitary Evolution Recurrent Neural Networks
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Invertible Residual Networks
• Fashionable Modelling with Flux
• Forward-Mode Automatic Differentiation in Julia
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• TensorNetwork: A Library for Physics and Machine Learning
• Don't Unroll Adjoint: Differentiating SSA-Form Programs
• Tropical Tensor Network for Ground States of Spin Glasses
• Mechanical Computing Systems Using Only Links and Rotary Joints
• Automatic differentiation of dominant eigensolver and its applications in quantum physics
• Back to the Future: The Case for Reversible Computing
• Reversible Recurrent Neural Networks
• Auto-Differentiating Linear Algebra
• Design and Implementation of a Reversible Object-Oriented Programming Language
• A Space-Efficient Design for Reversible Floating Point Adder in Quantum Computing
• Use and implementation of autodifferentiation in tensor network methods with complex scalars
• Automatic Differentiation for Complex Valued SVD
• Quantum circuits for floating-point arithmetic