FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

Abstract

A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network. Likelihood-based training of these models requires restricting their architectures to allow cheap computation of Jacobian determinants. Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation. In this paper, we use Hutchinson's trace estimator to give a scalable unbiased estimate of the log-density. The result is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures. We demonstrate our approach on high-dimensional density estimation, image generation, and variational inference, achieving the state-of-the-art among exact likelihood methods with efficient sampling.

8 Pages, 6 figures

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Spectral Normalization for Generative Adversarial Networks
• NICE: Non-linear Independent Components Estimation
• Neural Ordinary Differential Equations
• MADE: Masked Autoencoder for Distribution Estimation
• Sylvester Normalizing Flows for Variational Inference
• Transformation Autoregressive Networks

Cited by in corpus (79)

• Mining gold from implicit models to improve likelihood-free inference
• Invertible Residual Networks
• Denoising Diffusion Implicit Models
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
• Dissecting Neural ODEs
• Mining for Dark Matter Substructure: Inferring subhalo population properties from strong lenses with machine learning
• Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities
• Graph Neural Ordinary Differential Equations
• WaveFlow: A Compact Flow-based Model for Raw Audio
• Augmented Neural ODEs
• SoftFlow: Probabilistic Framework for Normalizing Flow on Manifolds
• Equivariant Flows: sampling configurations for multi-body systems with symmetric energies
• On Robustness of Neural Ordinary Differential Equations
• Riemannian Continuous Normalizing Flows
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
• A Tutorial on Deep Latent Variable Models of Natural Language
• Continuous-in-Depth Neural Networks
• Learning Neural Event Functions for Ordinary Differential Equations
• Learning to Encode Position for Transformer with Continuous Dynamical Model
• Diffusion Probabilistic Models for 3D Point Cloud Generation
• Hypersolvers: Toward Fast Continuous-Depth Models
• Stochastic Normalizing Flows
• STEER: Simple Temporal Regularization For Neural ODEs
• Euclideanizing Flows: Diffeomorphic Reduction for Learning Stable Dynamical Systems
• Robust learning with implicit residual networks
• Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies
• ShapeFlow: Learnable Deformations Among 3D Shapes
• TorchDyn: A Neural Differential Equations Library
• Training Invertible Neural Networks as Autoencoders
• TraDE: Transformers for Density Estimation
• MALI: A memory efficient and reverse accurate integrator for Neural ODEs
• Globally Injective ReLU Networks
• MintNet: Building Invertible Neural Networks with Masked Convolutions
• Temporal Normalizing Flows
• Latent Variable Modelling with Hyperbolic Normalizing Flows
• WaveNODE: A Continuous Normalizing Flow for Speech Synthesis
• Stochasticity in Neural ODEs: An Empirical Study
• On Scalable and Efficient Computation of Large Scale Optimal Transport
• Multimodal Generative Models for Compositional Representation Learning
• Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• Structured Output Learning with Conditional Generative Flows
• A Closer Look at Double Backpropagation
• Conditional Adversarial Generative Flow for Controllable Image Synthesis
• Path Integral Sampler: a stochastic control approach for sampling
• Trumpets: Injective Flows for Inference and Inverse Problems
• On the Need for Topology-Aware Generative Models for Manifold-Based Defenses
• Generative Hybrid Representations for Activity Forecasting with No-Regret Learning
• Non-Gaussian Gaussian Processes for Few-Shot Regression
• Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable Neural Distribution Alignment
• HyperFlow: Representing 3D Objects as Surfaces
• Probabilistic Software Modeling: A Data-driven Paradigm for Software Analysis
• Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics
• Relative gradient optimization of the Jacobian term in unsupervised deep learning
• Hypernetwork approach to generating point clouds
• Robust normalizing flows using Bernstein-type polynomials
• TreeFlow: Going beyond Tree-based Gaussian Probabilistic Regression
• Continuous-Depth Neural Models for Dynamic Graph Prediction
• Featurized Density Ratio Estimation
• MeshODE: A Robust and Scalable Framework for Mesh Deformation
• Video Extrapolation with an Invertible Linear Embedding
• Woodbury Transformations for Deep Generative Flows
• Second-Order Neural ODE Optimizer
• Modeling 3D Surface Manifolds with a Locally Conditioned Atlas
• Quasi-Autoregressive Residual (QuAR) Flows
• Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows
• STRODE: Stochastic Boundary Ordinary Differential Equation
• Noise Conditional Flow Model for Learning the Super-Resolution Space
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
• Autoencoding Under Normalization Constraints
• Arbitrary Conditional Distributions with Energy
• Robust Neural Networks inspired by Strong Stability Preserving Runge-Kutta methods
• NeurInt : Learning to Interpolate through Neural ODEs
• ADAVI: Automatic Dual Amortized Variational Inference Applied To Pyramidal Bayesian Models
• Trust the Critics: Generatorless and Multipurpose WGANs with Initial Convergence Guarantees
• RBM-Flow and D-Flow: Invertible Flows with Discrete Energy Base Spaces
• Particle Flow Bayes' Rule
• EBMs Trained with Maximum Likelihood are Generator Models Trained with a Self-adverserial Loss