Data Mining and Machine Learning in Astronomy

Abstract

We review the current state of data mining and machine learning in astronomy. 'Data Mining' can have a somewhat mixed connotation from the point of view of a researcher in this field. If used correctly, it can be a powerful approach, holding the potential to fully exploit the exponentially increasing amount of available data, promising great scientific advance. However, if misused, it can be little more than the black-box application of complex computing algorithms that may give little physical insight, and provide questionable results. Here, we give an overview of the entire data mining process, from data collection through to the interpretation of results. We cover common machine learning algorithms, such as artificial neural networks and support vector machines, applications from a broad range of astronomy, emphasizing those where data mining techniques directly resulted in improved science, and important current and future directions, including probability density functions, parallel algorithms, petascale computing, and the time domain. We conclude that, so long as one carefully selects an appropriate algorithm, and is guided by the astronomical problem at hand, data mining can be very much the powerful tool, and not the questionable black box.

Published in IJMPD. 61 pages, uses ws-ijmpd.cls. Several extra figures, some minor additions to the text

References in corpus (132)

• The Sloan Digital Sky Survey: Technical Summary
• Simulating the joint evolution of quasars, galaxies and their large-scale distribution
• The information bottleneck method
• Galaxy Zoo : Morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey
• First Results from the Catalina Real-time Transient Survey
• Bayesian photometric redshift estimation
• GAIA: Composition, Formation and Evolution of the Galaxy
• zCOSMOS: A Large VLT/VIMOS redshift survey covering 0 < z < 3 in the COSMOS field
• The Relationship Between Stellar Light Distributions of Galaxies and their Formation Histories
• Classifier Technology and the Illusion of Progress
• A concise reference to (projected) Sersic R^{1/n} quantities, including Concentration, Profile Slopes, Petrosian indices, and Kron Magnitudes
• ANNz: estimating photometric redshifts using artificial neural networks
• Photometric redshift and classification for the XMM-COSMOS sources
• The FIRST Bright Quasar Survey. II. 60 Nights and 1200 Spectra Later
• Efficient Photometric Selection of Quasars from the Sloan Digital Sky Survey: II. ~1,000,000 Quasars from Data Release Six
• Comparison of decision tree methods for finding active objects
• Slicing Through Multicolor Space: Galaxy Redshifts From Broadband Photometry
• Spectral Classification of Quasars in the Sloan Digital Sky Survey: Eigenspectra; Redshift and Luminosity Effects
• Science With The Australian Square Kilometre Array Pathfinder
• Evolution of the Galaxy Population Based on Photometric Redshifts in the Hubble Deep Field
• Photometric Redshifts for the SDSS Early Data Release
• The evolution of faint AGN between z~1 and z~5 from the COMBO-17 survey
• Star-forming galaxies at very high redshifts
• High Performance Direct Gravitational N-body Simulations on Graphics Processing Units -- II: An implementation in CUDA
• Spectral Classification of Galaxies: An Orthogonal Approach
• Estimating photometric redshifts with artificial neural networks
• Distributions of Galaxy Spectral Types in the Sloan Digital Sky Survey
• The Intrinsic Fraction of Broad Absorption Line Quasars
• A Galaxy Photometric Redshift Catalog for the Sloan Digital Sky Survey Data Release 6
• All-sky astrophysical component separation with Fast Independent Component Analysis (FastICA)
• Photometric Redshifts in the IRAC Shallow Survey
• Galaxy Types in the Sloan Digital Sky Survey Using Supervised Artificial Neural Networks
• Estimating the Redshift Distribution of Photometric Galaxy Samples II. Applications and Tests of a New Method
• Probabilistic Cross-Identification of Astronomical Sources
• A robust morphological classification of high-redshift galaxies using support vector machines on seeing limited images. I Method description
• SAPPORO: A way to turn your graphics cards into a GRAPE-6
• Photometric redshifts with the Multilayer Perceptron Neural Network: application to the HDF-S and SDSS
• Estimating Photometric Redshifts Using Support Vector Machines
• Automated Classification of Stellar Spectra. II: Two-Dimensional Classification with Neural Networks and Principal Components Analysis
• An Artificial Neural Network Approach to Classification of Galaxy Spectra
• AUTOMATED MORPHOLOGICAL CLASSIFICATION OF APM GALAXIES BY SUPERVISED ARTIFICIAL NEURAL NETWORKS
• Spectral Templates from Multicolor Redshift Surveys
• Creating Spectral Templates from Multicolor Redshift Surveys
• Photometric Redshifts of Quasars
• MegaZ-LRG: A photometric redshift catalogue of one million SDSS Luminous Red Galaxies
• Neural computation as a tool for galaxy classification : methods and examples
• Robust Machine Learning Applied to Astronomical Datasets I: Star-Galaxy Classification of the SDSS DR3 Using Decision Trees
• Galaxy Colour, Morphology, and Environment in the Sloan Digital Sky Survey
• Galaxies, Human Eyes and Artificial Neural Networks
• Initial Data Release from the INT Photometric H-alpha Survey of the Northern Galactic Plane (IPHAS)
• Robust Machine Learning Applied to Astronomical Datasets III: Probabilistic Photometric Redshifts for Galaxies and Quasars in the SDSS and GALEX
• The nature of the faint galaxies in the Hubble Deep Field
• A Robust Classification of Galaxy Spectra: Dealing with Noisy and Incomplete Data
• An Empirical Algorithm for Broad-band Photometric Redshifts of Quasars from the Sloan Digital Sky Survey
• ImpZ: a new photometric redshift code for galaxies and quasars
• Statistical Evidence for Three classes of Gamma-ray Bursts
• Ensemble Learning Independent Component Analysis of Normal Galaxy Spectra
• Fast Algorithms and Efficient Statistics: N-point Correlation Functions
• A Catalog of Color-based Redshift Estimates for z <~ 4 Galaxies in the Hubble Deep Field
• Wide Field Imaging. I. Applications of Neural Networks to object detection and star/galaxy classification
• Photometric redshifts from reconstructed QSO templates
• How to Find More Supernovae with Less Work: Object Classification Techniques for Difference Imaging
• Reconstructing Galaxy Spectral Energy Distributions from Broadband Photometry
• Genetic Algorithms and Supernovae Type Ia Analysis
• A Photometric Redshift Galaxy Catalog from the Red-Sequence Cluster Survey
• Automated Galaxy Morphology: A Fourier Approach
• Evolution of the Angular Correlation Function
• Photometric Redshifts for the Dark Energy Survey and VISTA and Implications for Large Scale Structure
• FERENGI: Redshifting galaxies from SDSS to GEMS, STAGES and COSMOS
• Towards More Precise Photometric Redshifts: Calibration Via CCD Photometry
• Robust Machine Learning Applied to Astronomical Datasets II: Quantifying Photometric Redshifts for Quasars Using Instance-Based Learning
• An Advanced, Three-Dimensional Plotting Library for Astronomy
• Bivariate Galaxy Luminosity Functions in the Sloan Digital Sky Survey
• Objective Classification of Galaxy Spectra using the Information Bottleneck Method
• Incorporating Photometric Redshift Probability Density Information into Real-Space Clustering Measurements
• Mining the SDSS archive. I. Photometric redshifts in the nearby universe
• A Census of Object Types and Redshift Estimates in the SDSS Photometric Catalog from a Trained Decision-Tree Classifier
• Using Oblique Decision Trees for the Morphological Classification of Galaxies
• What Lies Beneath: Using p(z) to Reduce Systematic Photometric Redshift Errors
• Revealing components of the galaxy population through nonparametric techniques
• Bayesian photometric redshifts with empirical training sets
• The Palomar-Quest Digital Synoptic Sky Survey
• Empirical Photometric Redshifts of Luminous Red Galaxies and Clusters in SDSS
• Finding rare objects and building pure samples: Probabilistic quasar classification from low resolution Gaia spectra
• A difference boosting neural network for automated star-galaxy classification
• Automated Probabilistic Classification of Transients and Variables
• A Unified Framework for Photometric Redshifts
• A robust morphological classification of high-redshift galaxies using support vector machines on seeing limited images. II. Quantifying morphological k-correction in the COSMOS field at 1<z<2: Ks band vs. I band
• Astrometric Redshifts for Quasars
• Background-Source Separation in astronomical images with Bayesian probability theory (I): the method
• A critical appraisal of the SED fitting method to estimate photometric redshifts
• Sunyaev-Zel'dovich clusters reconstruction in multiband bolometer camera surveys
• A reliable cluster detection technique using photometric redshifts: introducing the 2TecX algorithm
• Visualization, Exploration and Data Analysis of Complex Astrophysical Data
• Quantitative Morphology of Moderate Redshift Galaxies : How Many Peculiars are There ?
• Color-redshift Relations and Photometric Redshift Estimations of Quasars in Large Sky Surveys
• Automated Classification of Sloan Digital Sky Survey (SDSS) Stellar Spectra using Artificial Neural Networks
• Towards a library of synthetic galaxy spectra and preliminary results of classification and parametrization of unresolved galaxies for Gaia
• Two novel approaches for photometric redshift estimation based on SDSS and 2MASS databases
• Correlating galaxy morphologies and spectra in the 2dFGRS
• Use of neural networks for the identification of new z>=3.6 QSOs from FIRST-SDSS DR5
• Virtual Laboratories and Virtual Worlds
• Automated Classification of 2000 Bright IRAS Sources
• Matching Catalogues by Probabilistic Pattern Classification
• The Hubble Space Telescope WFPC2 B-Band Parallel Survey: A Study of Galaxy Morphology for 18 < B < 27 mag
• The Palomar Digital Sky Survey (DPOSS)
• An Active Instance-based Machine Learning method for Stellar Population Studies
• Improving Photometric Redshifts using GALEX Observations for the SDSS Stripe 82 and the Next Generation of SZ Cluster Surveys
• Photometric Redshifts and Signal-to-Noise
• Morphological Classification of Galaxies by Shapelet Decomposition in the Sloan Digital Sky Survey II: Multiwavelength Classification
• On the use of photometric redshifts for X-ray selected AGNs
• New Approaches to Object Classification in Synoptic Sky Surveys
• An introduction to artificial neural networks
• Morphology Classification and Photometric Redshift Measurement of Galaxies
• Developing and Deploying Advanced Algorithms to Novel Supercomputing Hardware
• Class Discovery in Galaxy Classification
• Interchanging Interactive 3-d Graphics for Astronomy
• Predicting spectral features in galaxy spectra from broad-band photometry
• Parametrization and Classification of 20 Billion LSST Objects: Lessons from SDSS
• Multi-Tree Methods for Statistics on Very Large Datasets in Astronomy
• Transient Detection and Classification
• Application of Medical Imaging Software to 3D Visualization of Astronomical Data
• Selection of quasar candidates from combined radio and optical surveys using neural networks
• Decision table for classifying point sources based on FIRST and 2MASS databases
• Searches for Rare and New Types of Objects
• Parameterisation of Galaxy Spectra in the 2dF Galaxy Redshift Survey
• Automated stellar classification for large surveys: a review of methods and results
• Objective Subclass Determination of Sloan Digital Sky Survey Unknown Spectral Objects
• Physical processes behind the morphological Hubble sequence
• The ESO Nearby Abell Cluster Survey VI. Spatial distribution and kinematics of early- and late-type galaxies
• Machine learning techniques for astrophysical modelling and photometric redshift estimation of quasars in optical sky surveys
• Multi-wavelength data handling in current and future surveys: the possible role of Virtual Observatory

Cited by in corpus (147)

• Fermi LAT observations of cosmic-ray electrons from 7 GeV to 1 TeV
• A catalog of visual-like morphologies in the 5 CANDELS fields using deep-learning
• An introduction to domain adaptation and transfer learning
• Photometric Supernova Classification With Machine Learning
• Photometric redshift estimation via deep learning
• Quantum Machine Learning for Chemistry and Physics
• Randomized algorithms for matrices and data
• Surveying the reach and maturity of machine learning and artificial intelligence in astronomy
• The weirdest SDSS galaxies: results from an outlier detection algorithm
• Using Machine Learning for Discovery in Synoptic Survey Imaging
• Neural Networks for Modeling and Control of Particle Accelerators
• SKYNET: an efficient and robust neural network training tool for machine learning in astronomy
• The DAWES review 10: The impact of deep learning for the analysis of galaxy surveys
• Big Universe, Big Data: Machine Learning and Image Analysis for Astronomy
• Analysing the 21cm signal from the Epoch of Reionization with artificial neural networks
• A Parameter Estimation Method that Directly Compares Gravitational Wave Observations to Numerical Relativity
• cosmoabc: Likelihood-free inference via Population Monte Carlo Approximate Bayesian Computation
• Deep Learning Topological Invariants of Band Insulators
• "Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?
• Astronomia ex machina: a history, primer, and outlook on neural networks in astronomy
• Kernel PCA for type Ia supernovae photometric classification
• Extragalactic Radio Continuum Surveys and the Transformation of Radio Astronomy
• Identification of Low Surface Brightness Tidal Features in Galaxies Using Convolutional Neural Networks
• Separation of pulsar signals from noise with supervised machine learning algorithms
• Machine learning cosmological structure formation
• Data Mining for Gravitationally Lensed Quasars
• Machine Learning and Cosmological Simulations II: Hydrodynamical Simulations
• Anomaly detection in the Zwicky Transient Facility DR3
• DASH: Deep Learning for the Automated Spectral Classification of Supernovae and their Hosts
• Machine Learning Techniques for Stellar Light Curve Classification
• Nonparametric Conditional Density Estimation in a High-Dimensional Regression Setting
• Beyond Spheroids and Discs: Classifications of CANDELS Galaxy Structure at 1.4 < z < 2 via Principal Component Analysis
• Learning the Relationship between Galaxies Spectra and their Star Formation Histories using Convolutional Neural Networks and Cosmological Simulations
• Machine Learning and Cosmological Simulations I: Semi-Analytical Models
• Full-sky Cosmic Microwave Background Foreground Cleaning Using Machine Learning
• Automated Classification of Periodic Variable Stars detected by the Wide-field Infrared Survey Explorer
• Machine learning technique for morphological classification of galaxies from the SDSS. I. Photometry-based approach
• Conditional Density Estimation Tools in Python and R with Applications to Photometric Redshifts and Likelihood-Free Cosmological Inference
• A review of unsupervised learning in astronomy
• A machine learning approach to mapping baryons onto dark matter haloes using the EAGLE and C-EAGLE simulations
• Astroinformatics of galaxies and quasars: a new general method for photometric redshifts estimation
• Detecting outliers and learning complex structures with large spectroscopic surveys - a case study with APOGEE stars
• Anomaly Detection in the Open Supernova Catalog
• An application of machine learning techniques to galaxy cluster mass estimation using the MACSIS simulations
• Machine learning classification of CHIME fast radio bursts -- I. Supervised methods
• Machine-assisted discovery of relationships in astronomy
• Evaluating the optical classification of Fermi BCUs using machine learning
• Supervised detection of exoplanets in high-contrast imaging sequences
• Multivariate Approaches to Classification in Extragalactic Astronomy
• Deep Learning Based Detection of Cosmological Diffuse Radio Sources
• An investigation on the factors affecting machine learning classifications in $\gamma$-ray astronomy
• Selecting Quasar Candidates by a SVM Classification System
• Exploring the spectroscopic diversity of type Ia supernovae with DRACULA: a machine learning approach
• Galaxy Morphology Classification using Neural Ordinary Differential Equations
• Robust PCA and MIC statistics of baryons in early mini-haloes
• Classifying Kepler light curves for 12,000 A and F stars using supervised feature-based machine learning
• Light curve classification with recurrent neural networks for GOTO: dealing with imbalanced data
• Estimating Photometric Redshifts of Quasars via K-nearest Neighbor Approach Based on Large Survey Databases
• Applications of Machine Learning to Predicting Core-collapse Supernova Explosion Outcomes
• Gravity Spy: Lessons Learned and a Path Forward
• Sky Surveys
• Machine Learning applied to asteroid dynamics
• A Unified Framework for Constructing, Tuning and Assessing Photometric Redshift Density Estimates in a Selection Bias Setting
• SUPPNet: Neural network for stellar spectrum normalisation
• GERLUMPH Data Release 2: 2.5 billion simulated microlensing light curves
• Photometric Classification of quasars from RCS-2 using Random Forest
• Inferring physical properties of galaxies from their emission line spectra
• A machine-learning photometric classifier for massive stars in nearby galaxies I. The method
• Classification of Fermi-LAT sources with deep learning using energy and time spectra
• Evaluating the classification of Fermi BCUs from the 4FGL Catalog Using Machine Learning
• ODUSSEAS: Upgraded version with new reference scale and parameter determinations for 82 planet-host M dwarf stars in SWEET-Cat
• Searching for AGN and Pulsar Candidates in 4FGL Unassociated Sources Using Machine Learning
• Population mixtures and searches of lensed and extended quasars across photometric surveys
• Shaping physical properties of galaxy subtypes in the VIPERS survey: environment matters
• High-Cadence Imaging and Imaging Spectroscopy at the GREGOR Solar Telescope - A Collaborative Research Environment for High-Resolution Solar Physics
• Detecting Quasars in Large-Scale Astronomical Surveys
• Deep Learning in Searching the Spectroscopic Redshift of Quasars
• Synergies between low- and intermediate-redshift galaxy populations revealed with unsupervised machine learning
• Density Based Outlier Scoring on Kepler Data
• Classification of Multiwavelength Transients with Machine Learning
• Protostellar classification using supervised machine learning algorithms
• Estimating Spectroscopic Redshifts by Using k Nearest Neighbors Regression I. Description of Method and Analysis
• Classifying surface probe images in strongly correlated electronic systems via machine learning
• Bayesian sparse reconstruction: a brute-force approach to astronomical imaging and machine learning
• NIR Tully-Fisher in the Zone of Avoidance. - II. 21 cm HI-line spectra of southern ZOA galaxies
• Exploring X-ray variability with unsupervised machine learning I. Self-organizing maps applied to XMM-Newton data
• Variable classification in the LSST era: Exploring a model for quasi-periodic light curves
• Machine Learning applied to Multifrequency Data in Astrophysics: Blazar Classification
• Searching for TeV Candidates in 4LAC High-synchrotron-peaked Frequency BL Lac Objects
• Random Forest classification of Gaia DR3 white dwarf-main sequence spectra: a feasibility study
• Analyzing the Galactic pulsar distribution with machine learning
• Fireball streak detection with minimal CPU processing requirements for the Desert Fireball Network data processing pipeline
• Enhanced Astronomical Source Classification with Integration of Attention Mechanisms and Vision Transformers
• Scientific Data Mining in Astronomy
• Improving galaxy morphology with machine learning
• Mapping the X-ray variability of GRS1915+105 with machine learning
• Flashes in a Star Stream: Automated Classification of Astronomical Transient Events
• Applications of Machine-Learning Algorithms for Infrared Colour Selection of Galactic Wolf-Rayet Stars
• Understanding of the properties of neural network approaches for transient light curve approximations
• CLaSPS: a new methodology for Knowledge extraction from complex astronomical dataset
• Are tiled display walls needed for astronomy?
• Photo-$z$ with CuBAN$z$: An improved photometric redshift estimator using Clustering aided Back Propagation Neural network
• Advanced Classification of Hot Subdwarf Binaries Using Artificial Intelligence Techniques and Gaia DR3 data
• Source finding, parametrization and classification for the extragalactic Effelsberg-Bonn HI Survey
• Ask The Machine: Systematic detection of wind-type outflows in low-mass X-ray binaries
• Selection of powerful radio galaxies with machine learning
• A test of the evolution of gas depletion factor in galaxy clusters using strong gravitational lensing systems
• Machine Learning for Searching the Dark Energy Survey for Trans-Neptunian Objects
• Stellar Cluster Detection using GMM with Deep Variational Autoencoder
• Exploring TeV candidates of Fermi blazars through machine learning
• Thresholds of descending algorithms in inference problems
• CzSL: Learning from citizen science, experts and unlabelled data in astronomical image classification
• FNet II: Spectral Classification of Quasars, Galaxies, Stars, and broad absorption line (BAL) Quasars
• On the application of dimensionality reduction and clustering algorithms for the classification of kinematic morphologies of galaxies
• Supernova Light Curves Approximation based on Neural Network Models
• Automated classification of Chandra X-ray point sources using machine learning methods
• A Hybrid method of accurate classification for Blazars Of Uncertain Type in Fermi LAT Catalogs
• PIPS, an advanced platform for period detection in time series -- I. Fourier-likelihood periodogram and application to RR Lyrae Stars
• Study on Outliers in the Big Stellar Spectral Dataset of the Fifth Data Release (DR5) of the Large Sky Area Multi-Object Fiber Spectroscopic Telescope (LAMOST)
• Sports stars: analyzing the performance of astronomers at visualization-based discovery
• Cosmic Ray Rejection with Attention Augmented Deep Learning
• Search for glitches of gamma-ray pulsars with deep learning
• Clusternets: A deep learning approach to probe clustering dark energy
• A convolutional-neural-network estimator of CMB constraints on dark matter energy injection
• Optical Spectroscopy with the Technology of Virtual Observatory
• Visualization in Astrophysics: Developing New Methods, Discovering Our Universe, and Educating the Earth
• Optimizing Supernova Classification with Interpretable Machine Learning Models
• AMADA-Analysis of Multidimensional Astronomical Datasets
• The Complexity and Information Content of Simulated Universes
• The VIMOS Public Extragalactic Redshift Survey (VIPERS). The complexity of galaxy populations at 0.4< z<1.3 revealed with unsupervised machine-learning algorithms
• Drifting Features: Detection and evaluation in the context of automatic RRLs identification in VVV
• Identifying Radio Active Galactic Nuclei with Machine Learning and Large-Area Surveys
• Construction of a far ultraviolet all sky map from an incomplete survey: Application of a deep learning algorithm
• SHAPE. I. A SOM-SED hybrid approach for efficient galaxy parameter estimation leveraging JWST
• Success of Machine Learning algorithms in Dynamical Mass Measurements of Galaxy Clusters
• Deep Learning Transient Detection with VERITAS
• Advanced Astroinformatics for Variable Star Classification
• A machine learning approach to estimate mid-infrared fluxes from WISE data
• A catalog of photometric redshift and the distribution of broad galaxy morphologies
• A test of Amati relation using HII galaxy distances
• The Taiwan Extragalactic Astronomical Data Center (TWEA-DC)
• Discussion on "Techniques for Massive-Data Machine Learning in Astronomy" by A. Gray
• Utilizing Astroinformatics to Maximize the Science Return of the Next Generation Virgo Cluster Survey
• Classification of Pulsars using Extreme Deconvolution
• Virtual Realities: Is there only one advanced image display that astronomers need?
• Quantifying Radio Source Morphology
• Automated Quantitative Description of Spiral Galaxy Arm-Segment Structure