Flow-based Generative Models for Learning Manifold to Manifold Mappings

Abstract

Many measurements or observations in computer vision and machine learning manifest as non-Euclidean data. While recent proposals (like spherical CNN) have extended a number of deep neural network architectures to manifold-valued data, and this has often provided strong improvements in performance, the literature on generative models for manifold data is quite sparse. Partly due to this gap, there are also no modality transfer/translation models for manifold-valued data whereas numerous such methods based on generative models are available for natural images. This paper addresses this gap, motivated by a need in brain imaging -- in doing so, we expand the operating range of certain generative models (as well as generative models for modality transfer) from natural images to images with manifold-valued measurements. Our main result is the design of a two-stream version of GLOW (flow-based invertible generative models) that can synthesize information of a field of one type of manifold-valued measurements given another. On the theoretical side, we introduce three kinds of invertible layers for manifold-valued data, which are not only analogous to their functionality in flow-based generative models (e.g., GLOW) but also preserve the key benefits (determinants of the Jacobian are easy to calculate). For experiments, on a large dataset from the Human Connectome Project (HCP), we show promising results where we can reliably and accurately reconstruct brain images of a field of orientation distribution functions (ODF) from diffusion tensor images (DTI), where the latter has a $5\times$ faster acquisition time but at the expense of worse angular resolution.

arXiv commentThis paper has been accepted by AAAI 2021. A video introduction is on YouTube: https://youtu.be/0r96U0vXsCM The official GitHub repo is: https://github.com/zhenxingjian/Dual_Manifold_GLOW

References in corpus (15)

• Auto-Encoding Variational Bayes
• Semi-Supervised Classification with Graph Convolutional Networks
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Geometric deep learning: going beyond Euclidean data
• Tutorial on Variational Autoencoders
• Variational Inference with Normalizing Flows
• Data Augmentation Generative Adversarial Networks
• Density estimation using Real NVP
• On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
• A Riemannian Network for SPD Matrix Learning
• Building Deep Networks on Grassmann Manifolds
• Flows for simultaneous manifold learning and density estimation
• ManifoldNet: A Deep Network Framework for Manifold-valued Data
• A CNN for homogneous Riemannian manifolds with applications to Neuroimaging
• DUAL-GLOW: Conditional Flow-Based Generative Model for Modality Transfer