Enhancing approximation abilities of neural networks by training derivatives

Abstract

A method to increase the precision of feedforward networks is proposed. It requires a prior knowledge of a target function derivatives of several orders and uses this information in gradient based training. Forward pass calculates not only the values of the output layer of a network but also their derivatives. The deviations of those derivatives from the target ones are used in an extended cost function and then backward pass calculates the gradient of the extended cost with respect to weights, which can then be used by any weights update algorithm. Despite a substantial increase in arithmetic operations per pattern (if compared to the conventional training), the extended cost allows to obtain 140--1000 times more accurate approximation for simple cases if the total number of operations is equal. This precision also happens to be out of reach for the regular cost function. The method fits well into the procedure of solving differential equations with neural networks. Unlike training a network to match some target mapping, which requires an explicit use of the target derivatives in the extended cost function, the cost function for solving a differential equation is based on the deviation of the equation's residual from zero and thus can be extended by differentiating the equation itself, which does not require any prior knowledge. Solving an equation with such a cost resulted in 13 times more accurate result and could be done with 3 times larger grid step. GPU-efficient algorithm for calculating the gradient of the extended cost function is proposed.

References in corpus (7)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• A unified deep artificial neural network approach to partial differential equations in complex geometries
• All you need is a good init
• Batch-normalized Maxout Network in Network
• Efficient Implementation of a Higher-Order Language with Built-In AD
• Backpropagation generalized for output derivatives
• Avoiding overfitting of multilayer perceptrons by training derivatives

Cited by in corpus (4)

• An Energy Approach to the Solution of Partial Differential Equations in Computational Mechanics via Machine Learning: Concepts, Implementation and Applications
• Deep Ensembles vs. Committees for Uncertainty Estimation in Neural-Network Force Fields: Comparison and Application to Active Learning
• Deep Learning Aided Laplace Based Bayesian Inference for Epidemiological Systems
• Studying First Passage Problems using Neural Networks: A Case Study in the Slit-Well Microfluidic Device