Data Mining and Machine Learning in Astronomy

Abstract

We review the current state of data mining and machine learning in astronomy. 'Data Mining' can have a somewhat mixed connotation from the point of view of a researcher in this field. If used correctly, it can be a powerful approach, holding the potential to fully exploit the exponentially increasing amount of available data, promising great scientific advance. However, if misused, it can be little more than the black-box application of complex computing algorithms that may give little physical insight, and provide questionable results. Here, we give an overview of the entire data mining process, from data collection through to the interpretation of results. We cover common machine learning algorithms, such as artificial neural networks and support vector machines, applications from a broad range of astronomy, emphasizing those where data mining techniques directly resulted in improved science, and important current and future directions, including probability density functions, parallel algorithms, petascale computing, and the time domain. We conclude that, so long as one carefully selects an appropriate algorithm, and is guided by the astronomical problem at hand, data mining can be very much the powerful tool, and not the questionable black box.

Published in IJMPD. 61 pages, uses ws-ijmpd.cls. Several extra figures, some minor additions to the text

References in corpus (50)

• zCOSMOS: A Large VLT/VIMOS redshift survey covering 0 < z < 3 in the COSMOS field
• Classifier Technology and the Illusion of Progress
• Photometric redshift and classification for the XMM-COSMOS sources
• Science With The Australian Square Kilometre Array Pathfinder
• High Performance Direct Gravitational N-body Simulations on Graphics Processing Units -- II: An implementation in CUDA
• The Intrinsic Fraction of Broad Absorption Line Quasars
• A Galaxy Photometric Redshift Catalog for the Sloan Digital Sky Survey Data Release 6
• Photometric Redshifts in the IRAC Shallow Survey
• Probabilistic Cross-Identification of Astronomical Sources
• A robust morphological classification of high-redshift galaxies using support vector machines on seeing limited images. I Method description
• MegaZ-LRG: A photometric redshift catalogue of one million SDSS Luminous Red Galaxies
• Galaxy Colour, Morphology, and Environment in the Sloan Digital Sky Survey
• Robust Machine Learning Applied to Astronomical Datasets I: Star-Galaxy Classification of the SDSS DR3 Using Decision Trees
• Initial Data Release from the INT Photometric H-alpha Survey of the Northern Galactic Plane (IPHAS)
• Robust Machine Learning Applied to Astronomical Datasets III: Probabilistic Photometric Redshifts for Galaxies and Quasars in the SDSS and GALEX
• ImpZ: a new photometric redshift code for galaxies and quasars
• How to Find More Supernovae with Less Work: Object Classification Techniques for Difference Imaging
• FERENGI: Redshifting galaxies from SDSS to GEMS, STAGES and COSMOS
• An Advanced, Three-Dimensional Plotting Library for Astronomy
• Mining the SDSS archive. I. Photometric redshifts in the nearby universe
• What Lies Beneath: Using p(z) to Reduce Systematic Photometric Redshift Errors
• Revealing components of the galaxy population through nonparametric techniques
• Bayesian photometric redshifts with empirical training sets
• The Palomar-Quest Digital Synoptic Sky Survey
• Empirical Photometric Redshifts of Luminous Red Galaxies and Clusters in SDSS
• Astrometric Redshifts for Quasars
• A robust morphological classification of high-redshift galaxies using support vector machines on seeing limited images. II. Quantifying morphological k-correction in the COSMOS field at 1<z<2: Ks band vs. I band
• Background-Source Separation in astronomical images with Bayesian probability theory (I): the method
• Visualization, Exploration and Data Analysis of Complex Astrophysical Data
• Automated Classification of Sloan Digital Sky Survey (SDSS) Stellar Spectra using Artificial Neural Networks
• Towards a library of synthetic galaxy spectra and preliminary results of classification and parametrization of unresolved galaxies for Gaia
• Two novel approaches for photometric redshift estimation based on SDSS and 2MASS databases
• Virtual Laboratories and Virtual Worlds
• Use of neural networks for the identification of new z>=3.6 QSOs from FIRST-SDSS DR5
• Matching Catalogues by Probabilistic Pattern Classification
• Photometric Redshifts and Signal-to-Noise
• Improving Photometric Redshifts using GALEX Observations for the SDSS Stripe 82 and the Next Generation of SZ Cluster Surveys
• New Approaches to Object Classification in Synoptic Sky Surveys
• Morphology Classification and Photometric Redshift Measurement of Galaxies
• Developing and Deploying Advanced Algorithms to Novel Supercomputing Hardware
• Interchanging Interactive 3-d Graphics for Astronomy
• Predicting spectral features in galaxy spectra from broad-band photometry
• Parametrization and Classification of 20 Billion LSST Objects: Lessons from SDSS
• Multi-Tree Methods for Statistics on Very Large Datasets in Astronomy
• Transient Detection and Classification
• Application of Medical Imaging Software to 3D Visualization of Astronomical Data
• Decision table for classifying point sources based on FIRST and 2MASS databases
• Physical processes behind the morphological Hubble sequence
• Multi-wavelength data handling in current and future surveys: the possible role of Virtual Observatory
• Machine learning techniques for astrophysical modelling and photometric redshift estimation of quasars in optical sky surveys

Cited by in corpus (147)

• Fermi LAT observations of cosmic-ray electrons from 7 GeV to 1 TeV
• A catalog of visual-like morphologies in the 5 CANDELS fields using deep-learning
• An introduction to domain adaptation and transfer learning
• Photometric Supernova Classification With Machine Learning
• Photometric redshift estimation via deep learning
• Quantum Machine Learning for Chemistry and Physics
• Randomized algorithms for matrices and data
• Surveying the reach and maturity of machine learning and artificial intelligence in astronomy
• The weirdest SDSS galaxies: results from an outlier detection algorithm
• Using Machine Learning for Discovery in Synoptic Survey Imaging
• Neural Networks for Modeling and Control of Particle Accelerators
• SKYNET: an efficient and robust neural network training tool for machine learning in astronomy
• The DAWES review 10: The impact of deep learning for the analysis of galaxy surveys
• Big Universe, Big Data: Machine Learning and Image Analysis for Astronomy
• Analysing the 21cm signal from the Epoch of Reionization with artificial neural networks
• A Parameter Estimation Method that Directly Compares Gravitational Wave Observations to Numerical Relativity
• cosmoabc: Likelihood-free inference via Population Monte Carlo Approximate Bayesian Computation
• Deep Learning Topological Invariants of Band Insulators
• "Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?
• Astronomia ex machina: a history, primer, and outlook on neural networks in astronomy
• Kernel PCA for type Ia supernovae photometric classification
• Extragalactic Radio Continuum Surveys and the Transformation of Radio Astronomy
• Identification of Low Surface Brightness Tidal Features in Galaxies Using Convolutional Neural Networks
• Separation of pulsar signals from noise with supervised machine learning algorithms
• Machine learning cosmological structure formation
• Data Mining for Gravitationally Lensed Quasars
• Machine Learning and Cosmological Simulations II: Hydrodynamical Simulations
• Anomaly detection in the Zwicky Transient Facility DR3
• DASH: Deep Learning for the Automated Spectral Classification of Supernovae and their Hosts
• Machine Learning Techniques for Stellar Light Curve Classification
• Nonparametric Conditional Density Estimation in a High-Dimensional Regression Setting
• Beyond Spheroids and Discs: Classifications of CANDELS Galaxy Structure at 1.4 < z < 2 via Principal Component Analysis
• Learning the Relationship between Galaxies Spectra and their Star Formation Histories using Convolutional Neural Networks and Cosmological Simulations
• Machine Learning and Cosmological Simulations I: Semi-Analytical Models
• Full-sky Cosmic Microwave Background Foreground Cleaning Using Machine Learning
• Automated Classification of Periodic Variable Stars detected by the Wide-field Infrared Survey Explorer
• Machine learning technique for morphological classification of galaxies from the SDSS. I. Photometry-based approach
• Conditional Density Estimation Tools in Python and R with Applications to Photometric Redshifts and Likelihood-Free Cosmological Inference
• A review of unsupervised learning in astronomy
• A machine learning approach to mapping baryons onto dark matter haloes using the EAGLE and C-EAGLE simulations
• Astroinformatics of galaxies and quasars: a new general method for photometric redshifts estimation
• Detecting outliers and learning complex structures with large spectroscopic surveys - a case study with APOGEE stars
• Anomaly Detection in the Open Supernova Catalog
• An application of machine learning techniques to galaxy cluster mass estimation using the MACSIS simulations
• Machine learning classification of CHIME fast radio bursts -- I. Supervised methods
• Machine-assisted discovery of relationships in astronomy
• Evaluating the optical classification of Fermi BCUs using machine learning
• Supervised detection of exoplanets in high-contrast imaging sequences
• Multivariate Approaches to Classification in Extragalactic Astronomy
• Deep Learning Based Detection of Cosmological Diffuse Radio Sources
• An investigation on the factors affecting machine learning classifications in $\gamma$-ray astronomy
• Selecting Quasar Candidates by a SVM Classification System
• Exploring the spectroscopic diversity of type Ia supernovae with DRACULA: a machine learning approach
• Galaxy Morphology Classification using Neural Ordinary Differential Equations
• Robust PCA and MIC statistics of baryons in early mini-haloes
• Classifying Kepler light curves for 12,000 A and F stars using supervised feature-based machine learning
• Light curve classification with recurrent neural networks for GOTO: dealing with imbalanced data
• Estimating Photometric Redshifts of Quasars via K-nearest Neighbor Approach Based on Large Survey Databases
• Applications of Machine Learning to Predicting Core-collapse Supernova Explosion Outcomes
• Gravity Spy: Lessons Learned and a Path Forward
• Sky Surveys
• Machine Learning applied to asteroid dynamics
• A Unified Framework for Constructing, Tuning and Assessing Photometric Redshift Density Estimates in a Selection Bias Setting
• SUPPNet: Neural network for stellar spectrum normalisation
• GERLUMPH Data Release 2: 2.5 billion simulated microlensing light curves
• Photometric Classification of quasars from RCS-2 using Random Forest
• Inferring physical properties of galaxies from their emission line spectra
• A machine-learning photometric classifier for massive stars in nearby galaxies I. The method
• Classification of Fermi-LAT sources with deep learning using energy and time spectra
• Evaluating the classification of Fermi BCUs from the 4FGL Catalog Using Machine Learning
• ODUSSEAS: Upgraded version with new reference scale and parameter determinations for 82 planet-host M dwarf stars in SWEET-Cat
• Searching for AGN and Pulsar Candidates in 4FGL Unassociated Sources Using Machine Learning
• Population mixtures and searches of lensed and extended quasars across photometric surveys
• Shaping physical properties of galaxy subtypes in the VIPERS survey: environment matters
• High-Cadence Imaging and Imaging Spectroscopy at the GREGOR Solar Telescope - A Collaborative Research Environment for High-Resolution Solar Physics
• Detecting Quasars in Large-Scale Astronomical Surveys
• Deep Learning in Searching the Spectroscopic Redshift of Quasars
• Synergies between low- and intermediate-redshift galaxy populations revealed with unsupervised machine learning
• Density Based Outlier Scoring on Kepler Data
• Classification of Multiwavelength Transients with Machine Learning
• Protostellar classification using supervised machine learning algorithms
• Estimating Spectroscopic Redshifts by Using k Nearest Neighbors Regression I. Description of Method and Analysis
• Classifying surface probe images in strongly correlated electronic systems via machine learning
• Bayesian sparse reconstruction: a brute-force approach to astronomical imaging and machine learning
• NIR Tully-Fisher in the Zone of Avoidance. - II. 21 cm HI-line spectra of southern ZOA galaxies
• Exploring X-ray variability with unsupervised machine learning I. Self-organizing maps applied to XMM-Newton data
• Variable classification in the LSST era: Exploring a model for quasi-periodic light curves
• Machine Learning applied to Multifrequency Data in Astrophysics: Blazar Classification
• Searching for TeV Candidates in 4LAC High-synchrotron-peaked Frequency BL Lac Objects
• Random Forest classification of Gaia DR3 white dwarf-main sequence spectra: a feasibility study
• Analyzing the Galactic pulsar distribution with machine learning
• Fireball streak detection with minimal CPU processing requirements for the Desert Fireball Network data processing pipeline
• Enhanced Astronomical Source Classification with Integration of Attention Mechanisms and Vision Transformers
• Scientific Data Mining in Astronomy
• Improving galaxy morphology with machine learning
• Mapping the X-ray variability of GRS1915+105 with machine learning
• Flashes in a Star Stream: Automated Classification of Astronomical Transient Events
• Applications of Machine-Learning Algorithms for Infrared Colour Selection of Galactic Wolf-Rayet Stars
• Understanding of the properties of neural network approaches for transient light curve approximations
• CLaSPS: a new methodology for Knowledge extraction from complex astronomical dataset
• Are tiled display walls needed for astronomy?
• Photo-$z$ with CuBAN$z$: An improved photometric redshift estimator using Clustering aided Back Propagation Neural network
• Advanced Classification of Hot Subdwarf Binaries Using Artificial Intelligence Techniques and Gaia DR3 data
• Source finding, parametrization and classification for the extragalactic Effelsberg-Bonn HI Survey
• Ask The Machine: Systematic detection of wind-type outflows in low-mass X-ray binaries
• Selection of powerful radio galaxies with machine learning
• A test of the evolution of gas depletion factor in galaxy clusters using strong gravitational lensing systems
• Machine Learning for Searching the Dark Energy Survey for Trans-Neptunian Objects
• Stellar Cluster Detection using GMM with Deep Variational Autoencoder
• Exploring TeV candidates of Fermi blazars through machine learning
• Thresholds of descending algorithms in inference problems
• CzSL: Learning from citizen science, experts and unlabelled data in astronomical image classification
• FNet II: Spectral Classification of Quasars, Galaxies, Stars, and broad absorption line (BAL) Quasars
• On the application of dimensionality reduction and clustering algorithms for the classification of kinematic morphologies of galaxies
• Supernova Light Curves Approximation based on Neural Network Models
• Automated classification of Chandra X-ray point sources using machine learning methods
• A Hybrid method of accurate classification for Blazars Of Uncertain Type in Fermi LAT Catalogs
• PIPS, an advanced platform for period detection in time series -- I. Fourier-likelihood periodogram and application to RR Lyrae Stars
• Study on Outliers in the Big Stellar Spectral Dataset of the Fifth Data Release (DR5) of the Large Sky Area Multi-Object Fiber Spectroscopic Telescope (LAMOST)
• Sports stars: analyzing the performance of astronomers at visualization-based discovery
• Cosmic Ray Rejection with Attention Augmented Deep Learning
• Search for glitches of gamma-ray pulsars with deep learning
• Clusternets: A deep learning approach to probe clustering dark energy
• A convolutional-neural-network estimator of CMB constraints on dark matter energy injection
• Optical Spectroscopy with the Technology of Virtual Observatory
• Visualization in Astrophysics: Developing New Methods, Discovering Our Universe, and Educating the Earth
• Optimizing Supernova Classification with Interpretable Machine Learning Models
• AMADA-Analysis of Multidimensional Astronomical Datasets
• The Complexity and Information Content of Simulated Universes
• The VIMOS Public Extragalactic Redshift Survey (VIPERS). The complexity of galaxy populations at 0.4< z<1.3 revealed with unsupervised machine-learning algorithms
• Drifting Features: Detection and evaluation in the context of automatic RRLs identification in VVV
• Identifying Radio Active Galactic Nuclei with Machine Learning and Large-Area Surveys
• Construction of a far ultraviolet all sky map from an incomplete survey: Application of a deep learning algorithm
• SHAPE. I. A SOM-SED hybrid approach for efficient galaxy parameter estimation leveraging JWST
• Success of Machine Learning algorithms in Dynamical Mass Measurements of Galaxy Clusters
• Deep Learning Transient Detection with VERITAS
• Advanced Astroinformatics for Variable Star Classification
• A machine learning approach to estimate mid-infrared fluxes from WISE data
• A catalog of photometric redshift and the distribution of broad galaxy morphologies
• A test of Amati relation using HII galaxy distances
• The Taiwan Extragalactic Astronomical Data Center (TWEA-DC)
• Discussion on "Techniques for Massive-Data Machine Learning in Astronomy" by A. Gray
• Utilizing Astroinformatics to Maximize the Science Return of the Next Generation Virgo Cluster Survey
• Classification of Pulsars using Extreme Deconvolution
• Virtual Realities: Is there only one advanced image display that astronomers need?
• Quantifying Radio Source Morphology
• Automated Quantitative Description of Spiral Galaxy Arm-Segment Structure