AstroCLIP: A Cross-Modal Foundation Model for Galaxies

Abstract

We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification. Our approach to implementing AstroCLIP consists of two parts. First, we embed galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We then align the encoders using a contrastive loss. We apply our method to spectra from the Dark Energy Spectroscopic Instrument and images from its corresponding Legacy Imaging Survey. Overall, we find remarkable performance on all downstream tasks, even relative to supervised baselines. For example, for a task like photometric redshift prediction, we find similar performance to a specifically-trained ResNet18, and for additional tasks like physical property estimation (stellar mass, age, metallicity, and sSFR), we beat this supervised baseline by 19\% in terms of $R^2$. We also compare our results to a state-of-the-art self-supervised single-modal model for galaxy images, and find that our approach outperforms this benchmark by roughly a factor of two on photometric redshift estimation and physical property prediction in terms of $R^2$, while remaining roughly in-line in terms of morphology classification. Ultimately, our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra.

arXiv comment18 pages, accepted in Monthly Notices of the Royal Astronomical Society, Presented at the NeurIPS 2023 AI4Science Workshop

References in corpus (27)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• LSST: from Science Drivers to Reference Design and Anticipated Data Products
• On the Opportunities and Risks of Foundation Models
• Overview of the DESI Legacy Imaging Surveys
• DINOv2: Learning Robust Visual Features without Supervision
• DESI Bright Galaxy Survey: Final Target Selection, Design, and Validation
• iBOT: Image BERT Pre-Training with Online Tokenizer
• Galaxy Zoo DECaLS: Detailed Visual Morphology Measurements from Volunteers and Deep Learning for 314,000 Galaxies
• Masked Autoencoders Are Scalable Vision Learners
• Photometric redshifts from SDSS images using a Convolutional Neural Network
• ClimaX: A foundation model for weather and climate
• A Cookbook of Self-Supervised Learning
• The DAWES review 10: The impact of deep learning for the analysis of galaxy surveys
• Identifying Strong Lenses with Unsupervised Machine Learning using Convolutional Autoencoder
• Dimensionality Reduction of SDSS Spectra with Variational Autoencoders
• EVA-CLIP: Improved Training Techniques for CLIP at Scale
• SEED: Self-supervised Distillation For Visual Representation
• Detecting outliers in astronomical images with deep generative networks
• Spreading vectors for similarity search
• Autoencoding Galaxy Spectra I: Architecture
• The DESI PRObabilistic Value-Added Bright Galaxy Survey (PROVABGS) Mock Challenge
• Autoencoding Galaxy Spectra II: Redshift Invariance and Outlier Detection
• Mapping the Diversity of Galaxy Spectra with Deep Unsupervised Machine Learning
• Multiple Physics Pretraining for Physical Surrogate Models
• Estimating Galactic Distances From Images Using Self-supervised Representation Learning

Cited by in corpus (21)

• Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models
• pathfinder: A Semantic Framework for Literature Review and Knowledge Discovery in Astronomy
• Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Models
• PICZL: Image-based Photometric Redshifts for AGN
• Representation Learning for Time-Domain High-Energy Astrophysics: Discovery of Extragalactic Fast X-ray Transient XRT 200515
• SKATR: A Self-Supervised Summary Transformer for SKA
• AstroClearNet: Deep image prior for multi-frame astronomical image restoration
• SpecPT (Spectroscopy Pre-trained Transformer) Model for Extragalactic Spectroscopy: I. Architecture and Automated Redshift Measurement
• Deep learning-based astronomical multimodal data fusion: A comprehensive review
• Learning novel representations of variable sources from multi-modal $\text{Gaia}$ data via autoencoders
• The spatially-resolved effect of mergers on the stellar mass assembly of MaNGA galaxies
• Uncertainty estimation for time series classification: Exploring predictive uncertainty in transformer-based models for variable stars
• Insights on Galaxy Evolution from Interpretable Sparse Feature Networks
• Automated galaxy sizes in Euclid images using the Segment Anything Model
• Applying Vision Transformers on Spectral Analysis of Astronomical Objects
• Robustness of Neural Ratio and Posterior Estimators to Distributional Shifts for Population-Level Dark Matter Analysis in Strong Gravitational Lensing
• Large Language Models -- the Future of Fundamental Physics?
• Can AI Dream of Unseen Galaxies? Conditional Diffusion Model for Galaxy Morphology Augmentation
• Radio Galaxy Zoo: Morphological classification by Fanaroff-Riley designation using self-supervised pre-training
• Extracting latent representations from X-ray spectra. Classification, regression, and accretion signatures of Chandra sources
• A Value-added Physical Properties Catalog for Low-redshift Galaxies from DESI Legacy Imaging Surveys DR10