Guided Image Generation with Conditional Invertible Neural Networks

Abstract

In this work, we address the task of natural image generation guided by a conditioning input. We introduce a new architecture called conditional invertible neural network (cINN). The cINN combines the purely generative INN model with an unconstrained feed-forward network, which efficiently preprocesses the conditioning input into useful features. All parameters of the cINN are jointly optimized with a stable, maximum likelihood-based training procedure. By construction, the cINN does not experience mode collapse and generates diverse samples, in contrast to e.g. cGANs. At the same time our model produces sharp images since no reconstruction loss is required, in contrast to e.g. VAEs. We demonstrate these properties for the tasks of MNIST digit generation and image colorization. Furthermore, we take advantage of our bi-directional cINN architecture to explore and manipulate emergent properties of the latent space, such as changing the image style in an intuitive way.

References in corpus (14)

• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• NICE: Non-linear Independent Components Estimation
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• A Learned Representation For Artistic Style
• Density estimation using Real NVP
• Analyzing Inverse Problems with Invertible Neural Networks
• CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training
• Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
• Excessive Invariance Causes Adversarial Vulnerability
• Comparison of Maximum Likelihood and GAN-based training of Real NVPs
• Training Generative Reversible Networks

Cited by in corpus (60)

• Normalizing Flows for Probabilistic Modeling and Inference
• WINNet: Wavelet-inspired Invertible Network for Image Denoising
• Machine Learning and LHC Event Generation
• Towards Blind Watermarking: Combining Invertible and Non-invertible Mechanisms
• Solving inverse problems using conditional invertible neural networks
• Generative Networks for Precision Enthusiasts
• Targeting Multi-Loop Integrals with Neural Networks
• Learning Likelihoods with Conditional Normalizing Flows
• CURTAINs for your Sliding Window: Constructing Unobserved Regions by Transforming Adjacent Intervals
• Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
• MadNIS -- Neural Multi-Channel Importance Sampling
• Enhanced Sampling in the Age of Machine Learning: Algorithms and Applications
• Two Invertible Networks for the Matrix Element Method
• Inference of cosmic-ray source properties by conditional invertible neural networks
• ν-Flows: Conditional Neutrino Regression
• Stabilizing Invertible Neural Networks Using Mixture Models
• Stochastic Normalizing Flows for Inverse Problems: a Markov Chains Viewpoint
• What's Anomalous in LHC Jets?
• Returning CP-Observables to The Frames They Belong
• PatchNR: Learning from Very Few Images by Patch Normalizing Flow Regularization
• Controllable Data Generation by Deep Learning: A Review
• The Landscape of Unfolding with Machine Learning
• AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flows
• Wavelet Flow: Fast Training of High Resolution Normalizing Flows
• Modeling Lost Information in Lossy Image Compression
• Understanding and Mitigating Exploding Inverses in Invertible Neural Networks
• Normalizing Flows for High-Dimensional Detector Simulations
• iUNets: Fully invertible U-Nets with Learnable Up- and Downsampling
• Neural Invertible Variable-degree Optical Aberrations Correction
• Optimal, fast, and robust inference of reionization-era cosmology with the 21cmPIE-INN
• The Hidden Uncertainty in a Neural Networks Activations
• Invertible Denoising Network: A Light Solution for Real Noise Removal
• Neural Empirical Bayes: Source Distribution Estimation and its Applications to Simulation-Based Inference
• Efficient mapping of phase diagrams with conditional Boltzmann Generators
• Deep Normalization for Speaker Vectors
• Enhancing Image Rescaling using Dual Latent Variables in Invertible Neural Network
• Generative Invertible Quantum Neural Networks
• Learning to Plan Optimally with Flow-based Motion Planner
• A Method to Model Conditional Distributions with Normalizing Flows
• FINO: Flow-based Joint Image and Noise Model
• Conditional Normalizing Flows for Low-Dose Computed Tomography Image Reconstruction
• A Flow-Based Model for Conditional and Probabilistic Electricity Consumption Profile Generation and Prediction
• Generative Design of a Gas Turbine Combustor Using Invertible Neural Networks
• Representing Ambiguity in Registration Problems with Conditional Invertible Neural Networks
• Nonreversible MCMC from conditional invertible transforms: a complete recipe with convergence guarantees
• Invertible Zero-Shot Recognition Flows
• Learning Non-linear Wavelet Transformation via Normalizing Flow
• Application-driven Validation of Posteriors in Inverse Problems
• A Regularized Conditional GAN for Posterior Sampling in Image Recovery Problems
• Inverse Aerodynamic Design of Gas Turbine Blades using Probabilistic Machine Learning
• Invertible Image Rescaling
• Noise Conditional Flow Model for Learning the Super-Resolution Space
• Normalizing Flows as a Novel PDF Turbulence Model
• Taming Reversible Halftoning via Predictive Luminance
• Black-box Adversarial Example Generation with Normalizing Flows
• Prior Flow Variational Autoencoder: A density estimation model for Non-Intrusive Load Monitoring
• Generating Data Augmentation samples for Semantic Segmentation of Salt Bodies in a Synthetic Seismic Image Dataset
• Disentangling Noise from Images: A Flow-Based Image Denoising Neural Network
• Mitigating Generation Shifts for Generalized Zero-Shot Learning
• Uncertainty quantification for ptychography using normalizing flows