Active Learning to Overcome Sample Selection Bias: Application to Photometric Variable Star Classification
arXiv:1106.2832 · doi:10.1088/0004-637X/744/2/192
Abstract
Despite the great promise of machine-learning algorithms to classify and predict astrophysical parameters for the vast numbers of astrophysical sources and transients observed in large-scale surveys, the peculiarities of the training data often manifest as strongly biased predictions on the data of interest. Typically, training sets are derived from historical surveys of brighter, more nearby objects than those from more extensive, deeper surveys (testing data). This sample selection bias can cause catastrophic errors in predictions on the testing data because a) standard assumptions for machine-learned model selection procedures break down and b) dense regions of testing space might be completely devoid of training data. We explore possible remedies to sample selection bias, including importance weighting (IW), co-training (CT), and active learning (AL). We argue that AL---where the data whose inclusion in the training set would most improve predictions on the testing set are queried for manual follow-up---is an effective approach and is appropriate for many astronomical applications. For a variable star classification problem on a well-studied set of stars from Hipparcos and OGLE, AL is the optimal method in terms of error rate on the testing data, beating the off-the-shelf classifier by 3.4% and the other proposed methods by at least 3.0%. To aid with manual labeling of variable stars, we developed a web interface which allows for easy light curve visualization and querying of external databases. Finally, we apply active learning to classify variable stars in the ASAS survey, finding dramatic improvement in our agreement with the ACVS catalog, from 65.5% to 79.5%, and a significant increase in the classifier's average confidence for the testing set, from 14.6% to 42.9%, after a few AL iterations.
arXiv comment43 pages, 11 figures, submitted to ApJ
References in corpus (24)
- Galaxy Zoo : Morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey
- GAIA: Composition, Formation and Evolution of the Galaxy
- The CoRoT satellite in flight : description and performance
- ANNz: estimating photometric redshifts using artificial neural networks
- On Machine-Learned Classification of Variable Stars with Sparse and Noisy Time-Series Data
- Automated supervised classification of variable stars I. Methodology
- Random forest automated supervised classification of Hipparcos periodic variable stars
- Results from the Supernova Photometric Classification Challenge
- Galaxy Types in the Sloan Digital Sky Survey Using Supervised Artificial Neural Networks
- Optimal Time-Series Selection of Quasars
- A robust morphological classification of high-redshift galaxies using support vector machines on seeing limited images. I Method description
- Estimating Photometric Redshifts Using Support Vector Machines
- Reconstructing Redshift Distributions with Cross-Correlations: Tests and an Optimized Recipe
- OGLE small amplitude red giant variables in the Galactic Bar
- Eight-Dimensional Mid-Infrared/Optical Bayesian Quasar Selection
- Long Secondary Periods and Binarity in Red Giant Stars
- Mining the SDSS archive. I. Photometric redshifts in the nearby universe
- Quantifying Photometric Redshift Errors in the Absence of Spectroscopic Redshifts
- Semi-supervised Learning for Photometric Supernova Classification
- Photometric redshift estimation using Gaussian processes
- Statistical Classification Techniques for Photometric Supernova Typing
- Towards a library of synthetic galaxy spectra and preliminary results of classification and parametrization of unresolved galaxies for Gaia
- Photometric identification of blue horizontal branch stars
- Calibrating photometric redshift distributions with cross-correlations
Cited by in corpus (40)
- Galaxy Zoo: Probabilistic Morphology through Bayesian CNNs and Active Learning
- Automating Discovery and Classification of Transients and Variable Stars in the Synoptic Survey Era
- Automated Transient Identification in the Dark Energy Survey
- Machine Learning for the Zwicky Transient Facility
- Using Machine Learning for Discovery in Synoptic Survey Imaging
- K2 Variable Catalogue II: Machine Learning Classification of Variable Stars and Eclipsing Binaries in K2 Fields 0-4
- Fink, a new generation of broker for the LSST community
- Big Universe, Big Data: Machine Learning and Image Analysis for Astronomy
- The IPAC Image Subtraction and Discovery Pipeline for the intermediate Palomar Transient Factory
- Machine Learning-based Brokers for Real-time Classification of the LSST Alert Stream
- Construction of a Calibrated Probabilistic Classification Catalog: Application to 50k Variable Sources in the All-Sky Automated Survey
- Computational Intelligence Challenges and Applications on Large-Scale Astronomical Time Series Databases
- A Package for the Automated Classification of Periodic Variable Stars
- Astronomaly: Personalised Active Anomaly Detection in Astronomical Data
- Optimizing spectroscopic follow-up strategies for supernova photometric classification with active learning
- Star-Galaxy Classification in Multi-Band Optical Imaging
- Gamma-Ray Active Galactic Nucleus Type through Machine-Learning Algorithms
- Automated Classification of Periodic Variable Stars detected by the Wide-field Infrared Survey Explorer
- Active Anomaly Detection for time-domain discoveries
- A machine learned classifier for RR Lyrae in the VVV survey
- Automated classification of Hipparcos unsolved variables
- Discovery of 36 eclipsing EL CVn binaries found by the Palomar Transient Factory
- O-MedAL: Online Active Deep Learning for Medical Image Analysis
- The ZTF Source Classification Project: I. Methods and Infrastructure
- Supervised Ensemble Classification of Kepler Variable Stars
- Active deep learning method for the discovery of objects of interest in large spectroscopic surveys
- Searching for AGN and Pulsar Candidates in 4FGL Unassociated Sources Using Machine Learning
- Discovery of Bright Galactic R Coronae Borealis and DY Persei Variables: Rare Gems Mined from ACVS
- Modeling Light Curves for Improved Classification
- Optimizing Automated Classification of Periodic Variable Stars in New Synoptic Surveys
- Young Stellar Clusters Containing Massive Young Stellar Objects in the VVV Survey
- Searching for TeV Candidates in 4LAC High-synchrotron-peaked Frequency BL Lac Objects
- Active Learning for Computationally Efficient Distribution of Binary Evolution Simulations
- The Synthetic-Oversampling Method: Using Photometric Colors to Discover Extremely Metal-Poor Stars
- Comparative Analysis of a Transition Region Bright Point with a Blinker and Coronal Bright Point Using Multiple EIS Emission Lines
- Learning with many experts: model selection and sparsity
- Informative Bayesian model selection for RR Lyrae star classifiers
- Uncertainty estimation for time series classification: Exploring predictive uncertainty in transformer-based models for variable stars
- Informative regularization for a multi-layer perceptron RR Lyrae classifier under data shift
- The Variable Sky Through the OGLE Eye