ASTROMER: A transformer-based embedding for the representation of light curves

Abstract

Taking inspiration from natural language embeddings, we present ASTROMER, a transformer-based model to create representations of light curves. ASTROMER was pre-trained in a self-supervised manner, requiring no human-labeled data. We used millions of R-band light sequences to adjust the ASTROMER weights. The learned representation can be easily adapted to other surveys by re-training ASTROMER on new sources. The power of ASTROMER consists of using the representation to extract light curve embeddings that can enhance the training of other models, such as classifiers or regressors. As an example, we used ASTROMER embeddings to train two neural-based classifiers that use labeled variable stars from MACHO, OGLE-III, and ATLAS. In all experiments, ASTROMER-based classifiers outperformed a baseline recurrent neural network trained on light curves directly when limited labeled data was available. Furthermore, using ASTROMER embeddings decreases computational resources needed while achieving state-of-the-art results. Finally, we provide a Python library that includes all the functionalities employed in this work. The library, main code, and pre-trained weights are available at https://github.com/astromer-science

References in corpus (5)

• A recurrent neural network for classification of unevenly sampled variable stars
• Scalable End-to-end Recurrent Neural Network for Variable star classification
• Deep Neural Network Classifier for Variable Stars with Novelty Detection Capability
• Deep Attention-Based Supernovae Classification of Multi-Band Light-Curves
• The effect of phased recurrent units in the classification of multiple catalogs of astronomical lightcurves

Cited by in corpus (18)

• Astronomia ex machina: a history, primer, and outlook on neural networks in astronomy
• Towards an astronomical foundation model for stars with a Transformer-based model
• Persistent and occasional: searching for the variable population of the ZTF/4MOST sky using ZTF data release 11
• ATAT: Astronomical Transformer for time series And Tabular data
• Identifying Light-curve Signals with a Deep Learning Based Object Detection Algorithm. II. A General Light Curve Classification Framework
• TEGLIE: Transformer encoders as strong gravitational lens finders in KiDS
• PICZL: Image-based Photometric Redshifts for AGN
• Application of Convolutional Neural Networks to time domain astrophysics. 2D image analysis of OGLE light curves
• Deep learning-based astronomical multimodal data fusion: A comprehensive review
• Uncertainty estimation for time series classification: Exploring predictive uncertainty in transformer-based models for variable stars
• Learning novel representations of variable sources from multi-modal $\text{Gaia}$ data via autoencoders
• FALCO: a Foundation model of Astronomical Light Curves for time dOmain astronomy
• Leveraging pre-trained vision Transformers for multi-band photometric light curve classification
• Real-time Light Curve Classification Framework for the Wide Field Survey Telescope Using Modified Semi-supervised Variational Auto-Encoder
• STAR NRE: Solving supernova selection effects with set-based truncated auto-regressive neural ratio estimation
• Applying Vision Transformers on Spectral Analysis of Astronomical Objects
• Multivariate time series transformer embeddings for light curves of periodic variable stars
• Exploring Late Stellar Evolution in the Era of Large Surveys: Machine Learning Prospects for Hot Subdwarfs and White Dwarfs