Enforcing constraints for interpolation and extrapolation in Generative Adversarial Networks

Abstract

We suggest ways to enforce given constraints in the output of a Generative Adversarial Network (GAN) generator both for interpolation and extrapolation (prediction). For the case of dynamical systems, given a time series, we wish to train GAN generators that can be used to predict trajectories starting from a given initial condition. In this setting, the constraints can be in algebraic and/or differential form. Even though we are predominantly interested in the case of extrapolation, we will see that the tasks of interpolation and extrapolation are related. However, they need to be treated differently. For the case of interpolation, the incorporation of constraints is built into the training of the GAN. The incorporation of the constraints respects the primary game-theoretic setup of a GAN so it can be combined with existing algorithms. However, it can exacerbate the problem of instability during training that is well-known for GANs. We suggest adding small noise to the constraints as a simple remedy that has performed well in our numerical experiments. The case of extrapolation (prediction) is more involved. During training, the GAN generator learns to interpolate a noisy version of the data and we enforce the constraints. This approach has connections with model reduction that we can utilize to improve the efficiency and accuracy of the training. Depending on the form of the constraints, we may enforce them also during prediction through a projection step. We provide examples of linear and nonlinear systems of differential equations to illustrate the various constructions.

arXiv comment29 pages; v2 has major text revision/restructuring, includes results for the Lorenz system and has several more references

References in corpus (11)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• DGM: A deep learning algorithm for solving partial differential equations
• Solving high-dimensional partial differential equations using deep learning
• Neural Ordinary Differential Equations
• Why does deep and cheap learning work so well?
• Data-assisted reduced-order modeling of extreme events in complex dynamical systems
• Information-theoretic analysis of generalization capability of learning algorithms
• An exact mapping between the Variational Renormalization Group and Deep Learning
• Nonparametric forecasting of low-dimensional dynamical systems
• Physics-constrained, data-driven discovery of coarse-grained dynamics

Cited by in corpus (12)

• Enforcing Statistical Constraints in Generative Adversarial Networks for Modeling Chaotic Dynamical Systems
• Solving inverse-PDE problems with physics-aware neural networks
• Symplectic Neural Networks in Taylor Series Form for Hamiltonian Systems
• Coercing Machine Learning to Output Physically Accurate Results
• Incorporating physical constraints in a deep probabilistic machine learning framework for coarse-graining dynamical systems
• Encoding Invariances in Deep Generative Models
• Enforcing Deterministic Constraints on Generative Adversarial Networks for Emulating Physical Systems
• Deep Learning of Dynamic Subsurface Flow via Theory-guided Generative Adversarial Network
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling
• Enforcing constraints for time series prediction in supervised, unsupervised and reinforcement learning
• Accurate generation of stochastic dynamics based on multi-model Generative Adversarial Networks
• Pathology-Aware Generative Adversarial Networks for Medical Image Augmentation