Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN

Abstract

Decoding and reconstructing images from brain imaging data is a research area of high interest. Recent progress in deep generative neural networks has introduced new opportunities to tackle this problem. Here, we employ a recently proposed large-scale bi-directional generative adversarial network, called BigBiGAN, to decode and reconstruct natural scenes from fMRI patterns. BigBiGAN converts images into a 120-dimensional latent space which encodes class and attribute information together, and can also reconstruct images based on their latent vectors. We computed a linear mapping between fMRI data, acquired over images from 150 different categories of ImageNet, and their corresponding BigBiGAN latent vectors. Then, we applied this mapping to the fMRI activity patterns obtained from 50 new test images from 50 unseen categories in order to retrieve their latent vectors, and reconstruct the corresponding images. Pairwise image decoding from the predicted latent vectors was highly accurate (84%). Moreover, qualitative and quantitative assessments revealed that the resulting image reconstructions were visually plausible, successfully captured many attributes of the original images, and had high perceptual similarity with the original content. This method establishes a new state-of-the-art for fMRI-based natural image reconstruction, and can be flexibly updated to take into account any future improvements in generative models of natural scene images.

arXiv commentAccepted to IEEE IJCNN2020

References in corpus (6)

• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Adversarial Feature Learning
• Large Scale Adversarial Representation Learning
• Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks
• From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI
• Reconstructing Perceived Images from Brain Activity by Visually-guided Cognitive Representation and Adversarial Learning

Cited by in corpus (5)

• DREAM: Visual Decoding from Reversing Human Visual System
• Understanding Cognitive Fatigue from fMRI Scans with Self-supervised Learning
• NeuroSwift: A Lightweight Cross-Subject Framework for fMRI Visual Reconstruction of Complex Scenes
• HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusion
• More Than Meets the Eye: Self-Supervised Depth Reconstruction From Brain Activity