Anomaly detection for machine learning redshifts applied to SDSS galaxies
arXiv:1503.08214 · doi:10.1093/mnras/stv1551
Abstract
We present an analysis of anomaly detection for machine learning redshift estimation. Anomaly detection allows the removal of poor training examples, which can adversely influence redshift estimates. Anomalous training examples may be photometric galaxies with incorrect spectroscopic redshifts, or galaxies with one or more poorly measured photometric quantity. We select 2.5 million 'clean' SDSS DR12 galaxies with reliable spectroscopic redshifts, and 6730 'anomalous' galaxies with spectroscopic redshift measurements which are flagged as unreliable. We contaminate the clean base galaxy sample with galaxies with unreliable redshifts and attempt to recover the contaminating galaxies using the Elliptical Envelope technique. We then train four machine learning architectures for redshift analysis on both the contaminated sample and on the preprocessed 'anomaly-removed' sample and measure redshift statistics on a clean validation sample generated without any preprocessing. We find an improvement on all measured statistics of up to 80% when training on the anomaly removed sample as compared with training on the contaminated sample for each of the machine learning routines explored. We further describe a method to estimate the contamination fraction of a base data sample.
arXiv comment13 pages, 8 figures, 1 table, minor text updates to macth MNRAS accepted version
References in corpus (22)
- Scikit-learn: Machine Learning in Python
- The 2.5 m Telescope of the Sloan Digital Sky Survey
- SDSS-III: Massive Spectroscopic Surveys of the Distant Universe, the Milky Way Galaxy, and Extra-Solar Planetary Systems
- K-corrections and filter transformations in the ultraviolet, optical, and near infrared
- The u'g'r'i'z' Standard Star Network
- A Critical Assessment of Photometric Redshift Methods: A CANDELS Investigation
- The PRIsm MUlti-Object Survey (PRIMUS) I: Survey Overview and Characteristics
- TPZ : Photometric redshift PDFs and ancillary information by using prediction trees and random forests
- PHAT: PHoto-z Accuracy Testing
- The PRIsm MUlti-object Survey (PRIMUS). II. Data Reduction and Redshift Fitting
- Photometric redshift analysis in the Dark Energy Survey Science Verification data
- ArborZ: Photometric Redshifts Using Boosted Decision Trees
- On using angular cross-correlations to determine source redshift distributions
- Catastrophic photometric redshift errors: weak lensing survey requirements
- SOMz: photometric redshift PDFs with self organizing maps and random atlas
- Using Galaxy Two-point Correlation Functions to Determine the Redshift Distributions of Galaxies Binned by Photometric Redshift
- Unsupervised self-organised mapping: a versatile empirical tool for object selection, classification and redshift estimation in large surveys
- Using neural networks to estimate redshift distributions. An application to CFHTLenS
- Feature importance for machine learning redshifts applied to SDSS galaxies
- Exhausting the Information: Novel Bayesian Combination of Photometric Redshift PDFs
- Data augmentation for machine learning redshifts applied to SDSS galaxies
- Spectroscopic failures in photometric redshift calibration: cosmological biases and survey requirements
Cited by in corpus (21)
- Surveying the reach and maturity of machine learning and artificial intelligence in astronomy
- Anomaly detection in the Zwicky Transient Facility DR3
- Morpho-z: improving photometric redshifts with galaxy morphology
- Active Anomaly Detection for time-domain discoveries
- A comprehensive statistical study on gamma-ray bursts
- Data mining techniques on astronomical spectra data. II : Classification Analysis
- Photometric redshifts with machine learning, lights and shadows on a complex data science use case
- Stacking for machine learning redshifts applied to SDSS galaxies
- Autoencoding Galaxy Spectra II: Redshift Invariance and Outlier Detection
- Inferring physical properties of galaxies from their emission line spectra
- Band Selection from Hyperspectral Images Using Attention-based Convolutional Neural Networks
- Decoding Dark Matter Substructure without Supervision
- Data Deluge in Astrophysics: Photometric Redshifts as a Template Use Case
- Dimension Estimation Using Autoencoders
- Merged or monolithic? Using machine-learning to reconstruct the dynamical history of simulated star clusters
- Tuning target selection algorithms to improve galaxy redshift estimates
- Data challenges as a tool for time-domain astronomy
- FNet II: Spectral Classification of Quasars, Galaxies, Stars, and broad absorption line (BAL) Quasars
- The many flavours of photometric redshifts
- Spectroscopic observations of the machine-learning selected anomaly catalogue from the AllWISE Sky Survey
- Auto-outlier Fusion Technique for Chest X-ray classification with Multi-head Attention Mechanism