Deep learning-based astronomical multimodal data fusion: A comprehensive review

Abstract

With the rapid advancements in observational technologies and the widespread implementation of large-scale sky surveys, diverse electromagnetic wave data (e.g., optical and infrared) and non-electromagnetic wave data (e.g., gravitational waves) have become increasingly accessible. Astronomy has thus entered an unprecedented era of data abundance and complexity. Astronomers have long relied on unimodal data analysis to perceive the universe, but these efforts often provide only limited insights when confronted with the current massive and heterogeneous astronomical data. In this context, multimodal data fusion (MDF), as an emerging method, provides new opportunities to enhance the value of astronomical data and deepening the understanding of the universe by integrating information from different modalities. Recent progress in artificial intelligence (AI), particularly in deep learning (DL), has greatly accelerated the development of multimodal research in astronomy. Therefore, a timely review of this field is essential. This paper begins by discussing the motivation and necessity of astronomical MDF, followed by an overview of astronomical data sources and major data modalities. It then introduces representative DL models commonly used in astronomical multimodal studies, the general fusion process as well as various fusion strategies, emphasizing their characteristics, applicability, advantages, and limitations. Subsequently, the paper surveys existing astronomical multimodal studies and datasets. Finally, the discussion section synthesizes key findings, identifies potential challenges, and suggests promising directions for future research. By offering a structured overview and critical analysis, this review aims to inspire and guide researchers engaged in DL-based MDF in astronomy.

References in corpus (45)

• Advanced Virgo: a 2nd generation interferometric gravitational wave detector
• LSST: from Science Drivers to Reference Design and Anticipated Data Products
• Analytic Lightcurves for Planetary Transit Searches
• The Zwicky Transient Facility: System Overview, Performance, and First Results
• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification
• CALIFA, the Calar Alto Legacy Integral Field Area survey: I. Survey presentation
• Astrometric Calibration of the Sloan Digital Sky Survey
• The Pierre Auger Cosmic Ray Observatory
• The IceCube Neutrino Observatory: Instrumentation and Online Systems
• Spectral Classification and Redshift Measurement for the SDSS-III Baryon Oscillation Spectroscopic Survey
• ANNz: estimating photometric redshifts using artificial neural networks
• Multi-Messenger Astrophysics
• Classifying Radio Galaxies with Convolutional Neural Network
• Selection of radio pulsar candidates using artificial neural networks
• Resolving galaxies in time and space: II: Uncertainties in the spectral synthesis of datacubes
• Real-Time Detection of Gravitational Waves from Binary Neutron Stars using Artificial Neural Networks
• Enabling real-time multi-messenger astrophysics discoveries with deep learning
• On the average Gamma-Ray Burst X-ray flaring activity
• Experience with the Hubble Space Telescope: 20 years of an archetype
• AstroCLIP: A Cross-Modal Foundation Model for Galaxies
• Morphological classification of galaxies with deep learning: comparing 3-way and 4-way CNNs
• AstroVaDEr: Astronomical Variational Deep Embedder for Unsupervised Morphological Classification of Galaxies and Synthetic Image Generation
• Towards an astronomical foundation model for stars with a Transformer-based model
• Data augmentation for machine learning redshifts applied to SDSS galaxies
• Attention-gating for improved radio galaxy classification
• ASTROMER: A transformer-based embedding for the representation of light curves
• Light curve classification with recurrent neural networks for GOTO: dealing with imbalanced data
• Finding flares in Kepler and TESS data with recurrent deep neural networks
• PhotoRedshift-MML: a multimodal machine learning method for estimating photometric redshifts of quasars
• ATAT: Astronomical Transformer for time series And Tabular data
• A Multimodal Dataset and Benchmark for Radio Galaxy and Infrared Host Detection
• What's the Difference? The potential for Convolutional Neural Networks for transient detection without template subtraction
• Deep learning-based radiointerferometric imaging with GAN-aided training
• Periodic Variable Star Classification with Deep Learning: Handling Data Imbalance in an Ensemble Augmentation Way
• A Beginner's Guide to Working with Astronomical Data
• Deep Learning and LLM-based Methods Applied to Stellar Lightcurve Classification
• DeepGraviLens: a Multi-Modal Architecture for Classifying Gravitational Lensing Data
• Astronomical observations: a guide for allied researchers
• Galaxy Imaging with Generative Models: Insights from a Two-Models Framework
• X-ray Sources Classification Using Machine Learning: A Study with EP-WXT Pathfinder LEIA
• Denoising Diffusion Probabilistic Model for realistic and fast generated \textit{Euclid}-like data for weak lensing analysis
• Morpho-Photometric Classification of KiDS DR5 Sources Based on Neural Networks: A Comprehensive Star-Quasar-Galaxy Catalog
• Learning novel representations of variable sources from multi-modal $\text{Gaia}$ data via autoencoders
• FALCO: a Foundation model of Astronomical Light Curves for time dOmain astronomy

Cited by in corpus (1)

• Classification of Eclipsing Binary Light Curves in Gaia DR3: A Machine Learning Approach