A Differentiable Programming System to Bridge Machine Learning and Scientific Computing

Abstract

Scientific computing is increasingly incorporating the advancements in machine learning and the ability to work with large amounts of data. At the same time, machine learning models are becoming increasingly sophisticated and exhibit many features often seen in scientific computing, stressing the capabilities of machine learning frameworks. Just as the disciplines of scientific computing and machine learning have shared common underlying infrastructure in the form of numerical linear algebra, we now have the opportunity to further share new computational infrastructure, and thus ideas, in the form of Differentiable Programming. We describe Zygote, a Differentiable Programming system that is able to take gradients of general program structures. We implement this system in the Julia programming language. Our system supports almost all language constructs (control flow, recursion, mutation, etc.) and compiles high-performance code without requiring any user intervention or refactoring to stage computations. This enables an expressive programming model for deep learning, but more importantly, it enables us to incorporate a large ecosystem of libraries in our models in a straightforward way. We discuss our approach to automatic differentiation, including its support for advanced techniques such as mixed-mode, complex and checkpointed differentiation, and present several examples of differentiating programs.

Submitted to NeurIPS 2019

Cited by in corpus (15)

• Differentiable modeling to unify machine learning and physical models and advance Geosciences
• Learning neural network potentials from experimental data via Differentiable Trajectory Reweighting
• Multi-resolution partial differential equations preserved learning framework for spatiotemporal dynamics
• End-to-End Optimization of Metasurfaces for Imaging with Compressed Sensing
• Robust quantum gates using smooth pulses and physics-informed neural networks
• Differentiable Programming for Earth System Modeling
• A scheme for automatic differentiation of complex loss functions
• Learned multiphysics inversion with differentiable programming and machine learning
• How Well Does Kohn-Sham Regularizer Work for Weakly Correlated Systems?
• NeuralFMU: Towards Structural Integration of FMUs into Neural Networks
• Fault Prognosis of Turbofan Engines: Eventual Failure Prediction and Remaining Useful Life Estimation
• Fast Laser Cooling Using Optimal Quantum Control
• Deep learning and differential equations for modeling changes in individual-level latent dynamics between observation periods
• Algorithmic differentiation for plane-wave DFT: materials design, error control and learning model parameters
• E2V-SDE: From Asynchronous Events to Fast and Continuous Video Reconstruction via Neural Stochastic Differential Equations