Towards an astronomical foundation model for stars with a Transformer-based model

Abstract

Rapid strides are currently being made in the field of artificial intelligence using Transformer-based models like Large Language Models (LLMs). The potential of these methods for creating a single, large, versatile model in astronomy has not yet been explored. In this work, we propose a framework for data-driven astronomy that uses the same core techniques and architecture as used by LLMs. Using a variety of observations and labels of stars as an example, we build a Transformer-based model and train it in a self-supervised manner with cross-survey data sets to perform a variety of inference tasks. In particular, we demonstrate that a $\text{single}$ model can perform both discriminative and generative tasks even if the model was not trained or fine-tuned to do any specific task. For example, on the discriminative task of deriving stellar parameters from Gaia XP spectra, we achieve an accuracy of 47 K in $T_{\mathrm{eff}}$, 0.11 dex in $\log g$, and 0.07 dex in $[M/H]$, outperforming an expert $\text{XGBoost}$ model in the same setting. But the same model can also generate XP spectra from stellar parameters, inpaint unobserved spectral regions, extract empirical stellar loci, and even determine the interstellar extinction curve. Our framework demonstrates that building and training a $\text{single}$ foundation model without fine-tuning using data and parameters from multiple surveys to predict unmeasured observations and parameters is well within reach. Such "Large Astronomy Models" trained on large quantities of observational data will play a large role in the analysis of current and future large surveys.

References in corpus (50)

• Efficient Estimation of Word Representations in Vector Space
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Maps of Dust IR Emission for Use in Estimation of Reddening and CMBR Foregrounds
• Decoupled Weight Decay Regularization
• Measuring Reddening with SDSS Stellar Spectra and Recalibrating SFD
• The Gaia mission
• Deep Residual Learning for Image Recognition
• Gaia Data Release 3: Summary of the content and survey properties
• LSST: from Science Drivers to Reference Design and Anticipated Data Products
• Correcting for the Effects of Interstellar Extinction
• Gaussian Error Linear Units (GELUs)
• LoRA: Low-Rank Adaptation of Large Language Models
• The Apache Point Observatory Galactic Evolution Experiment (APOGEE)
• Sloan Digital Sky Survey IV: Mapping the Milky Way, Nearby Galaxies, and the Distant Universe
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• A 3D Dust Map Based on Gaia, Pan-STARRS 1 and 2MASS
• Emergent Abilities of Large Language Models
• Momentum Contrast for Unsupervised Visual Representation Learning
• Gaia Early Data Release 3: Parallax bias versus magnitude, colour, and position
• Mixed Precision Training
• Gaia Early Data Release 3: Photometric content and validation
• ASPCAP: The Apogee Stellar Parameter and Chemical Abundances Pipeline
• Modelling the Galactic Interstellar Extinction Distribution in Three Dimensions
• The COBE Diffuse Infrared Background Experiment Search for the Cosmic Infrared Background: I. Limits and Detections
• A three-dimensional Galactic extinction model
• The Apache Point Observatory Galactic Evolution Experiment (APOGEE) Spectrographs
• On Galactic density modeling in the presence of dust extinction
• Gaia Data Release 3: Processing and validation of BP/RP low-resolution spectral data
• Deep learning of multi-element abundances from high-resolution spectroscopic data
• Gaia Data Release 3: External calibration of BP/RP low-resolution spectroscopic data
• Simultaneous calibration of spectro-photometric distances and the Gaia DR2 parallax zero-point offset with deep learning
• Parameters of 220 million stars from Gaia BP/RP spectra
• Robust Data-driven Metallicities for 175 Million Stars from Gaia XP Spectra
• The Poor Old Heart of the Milky Way
• Internal calibration of Gaia BP/RP low-resolution spectra
• A data science platform to enable time-domain astronomy
• Astronomia ex machina: a history, primer, and outlook on neural networks in astronomy
• A direct measurement of the distance to the Galactic center using the kinematics of bar stars
• Mining for Strong Gravitational Lenses with Self-supervised Learning
• ASTROMER: A transformer-based embedding for the representation of light curves
• A variational encoder-decoder approach to precise spectroscopic age estimation for large Galactic surveys
• Improving \textsl{Gaia} parallax precision with a data-driven model of stars
• Deep Attention-Based Supernovae Classification of Multi-Band Light-Curves
• Hunting for C-rich long-period variable stars in the Milky Way's bar-bulge using unsupervised classification of Gaia BP/RP spectra
• Towards Galaxy Foundation Models with Hybrid Contrastive Learning
• Astroformer: More Data Might not be all you need for Classification
• Paying Attention to Astronomical Transients: Introducing the Time-series Transformer for Photometric Classification
• Radio Galaxy Zoo: Towards building the first multi-purpose foundation model for radio astronomy with self-supervised learning
• Toward a Spectral Foundation Model: An Attention-Based Approach with Domain-Inspired Fine-Tuning and Wavelength Parameterization
• Closing the stellar labels gap: An unsupervised, generative model for $\text{Gaia}$ BP/RP spectra

Cited by in corpus (9)

• The Three-Phase Evolution of the Milky Way
• Metallicity and $\alpha$-abundance for 48 million stars in low-extinction regions in the Milky Way
• Closing the stellar labels gap: Stellar label independent evidence for [$\alpha /M$] information in Gaia BP/RP spectra
• Mapping the Milky Way with Gaia Bp/Rp spectra I: Systematic flux corrections and atmospheric parameters for 68 million stars
• Mapping Synthetic Observations to Prestellar Core Models: An Interpretable Machine Learning Approach
• A Deep Neural Network Based Reverse Radio Spectrogram Search Algorithm
• Deep learning-based astronomical multimodal data fusion: A comprehensive review
• Exploring Galaxy Properties of eCALIFA with Contrastive Learning
• A CNN--Transformer Denoiser for low-$S/N$ Galaxy Spectra: Stellar Population Recovery in Synthetic Tests