AstroCLIP: A Cross-Modal Foundation Model for Galaxies

Abstract

We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification. Our approach to implementing AstroCLIP consists of two parts. First, we embed galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We then align the encoders using a contrastive loss. We apply our method to spectra from the Dark Energy Spectroscopic Instrument and images from its corresponding Legacy Imaging Survey. Overall, we find remarkable performance on all downstream tasks, even relative to supervised baselines. For example, for a task like photometric redshift prediction, we find similar performance to a specifically-trained ResNet18, and for additional tasks like physical property estimation (stellar mass, age, metallicity, and sSFR), we beat this supervised baseline by 19\% in terms of $R^2$. We also compare our results to a state-of-the-art self-supervised single-modal model for galaxy images, and find that our approach outperforms this benchmark by roughly a factor of two on photometric redshift estimation and physical property prediction in terms of $R^2$, while remaining roughly in-line in terms of morphology classification. Ultimately, our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra.

18 pages, accepted in Monthly Notices of the Royal Astronomical Society, Presented at the NeurIPS 2023 AI4Science Workshop

References in corpus (6)

• Masked Autoencoders Are Scalable Vision Learners
• EVA-CLIP: Improved Training Techniques for CLIP at Scale
• SEED: Self-supervised Distillation For Visual Representation
• Mapping the Diversity of Galaxy Spectra with Deep Unsupervised Machine Learning
• Multiple Physics Pretraining for Physical Surrogate Models
• Estimating Galactic Distances From Images Using Self-supervised Representation Learning

Cited by in corpus (21)

• Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models
• pathfinder: A Semantic Framework for Literature Review and Knowledge Discovery in Astronomy
• Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Models
• PICZL: Image-based Photometric Redshifts for AGN
• Representation Learning for Time-Domain High-Energy Astrophysics: Discovery of Extragalactic Fast X-ray Transient XRT 200515
• SKATR: A Self-Supervised Summary Transformer for SKA
• AstroClearNet: Deep image prior for multi-frame astronomical image restoration
• SpecPT (Spectroscopy Pre-trained Transformer) Model for Extragalactic Spectroscopy: I. Architecture and Automated Redshift Measurement
• Deep learning-based astronomical multimodal data fusion: A comprehensive review
• Learning novel representations of variable sources from multi-modal $\text{Gaia}$ data via autoencoders
• The spatially-resolved effect of mergers on the stellar mass assembly of MaNGA galaxies
• Uncertainty estimation for time series classification: Exploring predictive uncertainty in transformer-based models for variable stars
• Insights on Galaxy Evolution from Interpretable Sparse Feature Networks
• Automated galaxy sizes in Euclid images using the Segment Anything Model
• Applying Vision Transformers on Spectral Analysis of Astronomical Objects
• Robustness of Neural Ratio and Posterior Estimators to Distributional Shifts for Population-Level Dark Matter Analysis in Strong Gravitational Lensing
• Large Language Models -- the Future of Fundamental Physics?
• Can AI Dream of Unseen Galaxies? Conditional Diffusion Model for Galaxy Morphology Augmentation
• Radio Galaxy Zoo: Morphological classification by Fanaroff-Riley designation using self-supervised pre-training
• Extracting latent representations from X-ray spectra. Classification, regression, and accretion signatures of Chandra sources
• A Value-added Physical Properties Catalog for Low-redshift Galaxies from DESI Legacy Imaging Surveys DR10