DINOspec: Efficient Multimodal Alignment of Vision and Spectral Foundation Models for Astronomy

Abstract

Astronomical observations provide multimodal views of physical systems, with images and spectra capturing complementary properties of celestial objects. Scientific foundation models can learn powerful representations from these observations, but representations learned by separate models remain difficult to combine. We investigate whether laten space representations learned by separate vision and spectral models can be aligned without retraining their encoders. We introduce DINOspec, a multimodal framework that aligns a frozen DINOv3 image encoder with a pre-trained AION-1 spectral encoder using lightweight adapters and contrastive learning on 20,472 paired images and spectra of astronomical objects. DINOspec improves galaxy morphology classification (F1: 0.72$\to$0.78) and spectral type classification (F1: 0.70$\to$0.74) while training at few as 21M parameters. Redshift prediction shows no consistent improvement over the AION-1 baseline of $R^2\approx 0.9$, demonstrating that potential benefits of alignment depends on the configuration and downstream task. These results provide an empirical study of composing astronomical representations from independently pretrained foundation models through lightweight alignment modules.

arXiv comment5 pages, 1 figure, accepted to NeurIPS Representations for the Physical Sciences Workshop