Differentiable Spline Approximations

Abstract

The paradigm of differentiable programming has significantly enhanced the scope of machine learning via the judicious use of gradient-based optimization. However, standard differentiable programming methods (such as autodiff) typically require that the machine learning models be differentiable, limiting their applicability. Our goal in this paper is to use a new, principled approach to extend gradient-based optimization to functions well modeled by splines, which encompass a large family of piecewise polynomial models. We derive the form of the (weak) Jacobian of such functions and show that it exhibits a block-sparse structure that can be computed implicitly and efficiently. Overall, we show that leveraging this redesigned Jacobian in the form of a differentiable "layer" in predictive models leads to improved performance in diverse applications such as image segmentation, 3D point cloud reconstruction, and finite element analysis.

arXiv comment9 pages, accepted in Neurips 2021

References in corpus (13)

• Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations
• hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition
• Differentiable Convex Optimization Layers
• Estimating or Propagating Gradients Through Stochastic Neurons
• Differentiable Dynamic Programming for Structured Prediction and Attention
• DiffTaichi: Differentiable Programming for Physical Simulation
• Physics-informed neural networks (PINNs) for fluid mechanics: A review
• A differentiable programming method for quantum control
• Learning with Differentiable Perturbed Optimizers
• Learning to Control PDEs with Differentiable Physics
• Scalable Differentiable Physics for Learning and Control
• Accelerating PDE-constrained Inverse Solutions with Deep Learning and Reduced Order Models
• Differentiable Top-k Operator with Optimal Transport