Real-Time Data Mining of Massive Data Streams from Synoptic Sky Surveys

Abstract

The nature of scientific and technological data collection is evolving rapidly: data volumes and rates grow exponentially, with increasing complexity and information content, and there has been a transition from static data sets to data streams that must be analyzed in real time. Interesting or anomalous phenomena must be quickly characterized and followed up with additional measurements via optimal deployment of limited assets. Modern astronomy presents a variety of such phenomena in the form of transient events in digital synoptic sky surveys, including cosmic explosions (supernovae, gamma ray bursts), relativistic phenomena (black hole formation, jets), potentially hazardous asteroids, etc. We have been developing a set of machine learning tools to detect, classify and plan a response to transient events for astronomy applications, using the Catalina Real-time Transient Survey (CRTS) as a scientific and methodological testbed. The ability to respond rapidly to the potentially most interesting events is a key bottleneck that limits the scientific returns from the current and anticipated synoptic sky surveys. Similar challenge arise in other contexts, from environmental monitoring using sensor networks to autonomous spacecraft systems. Given the exponential growth of data rates, and the time-critical response, we need a fully automated and robust approach. We describe the results obtained to date, and the possible future developments.

14 pages, an invited paper for a special issue of Future Generation Computer Systems, Elsevier Publ. (2015). This is an expanded version of a paper arXiv:1407.3502 presented at the IEEE e-Science 2014 conf., with some new content

References in corpus (19)

• First Results from the Catalina Real-time Transient Survey
• Measuring the rotation period distribution of field M-dwarfs with Kepler
• A possible close supermassive black-hole binary in a quasar with optical periodicity
• A systematic search for close supermassive black hole binaries in the Catalina Real-Time Transient Survey
• On Machine-Learned Classification of Variable Stars with Sparse and Noisy Time-Series Data
• Random forest automated supervised classification of Hipparcos periodic variable stars
• Automating Discovery and Classification of Transients and Variable Stars in the Synoptic Survey Era
• A comparison of period finding algorithms
• Using conditional entropy to identify periodicity
• Improved AGN light curve analysis with the z-transformed discrete correlation function
• Machine-assisted discovery of relationships in astronomy
• Automated Probabilistic Classification of Transients and Variables
• Data challenges of time domain astronomy
• Sky Surveys
• Feature Selection Strategies for Classifying High Dimensional Astronomical Data Sets
• Towards an Automated Classification of Transient Events in Synoptic Sky Surveys
• New Approaches to Object Classification in Synoptic Sky Surveys
• Connecting the time domain community with the Virtual Astronomical Observatory
• Automated Real-Time Classification and Decision Making in Massive Data Streams from Synoptic Sky Surveys

Cited by in corpus (9)

• Machine Learning in Astronomy: a practical overview
• Surveying the reach and maturity of machine learning and artificial intelligence in astronomy
• A recurrent neural network for classification of unevenly sampled variable stars
• Deep-Learnt Classification of Light Curves
• Comparing Multi-class, Binary and Hierarchical Machine Learning Classification schemes for variable stars
• Efficient Multiple Incremental Computation for Kernel Ridge Regression with Bayesian Uncertainty Modeling
• Machine learning technique for morphological classification of galaxies from SDSS. II. The image-based morphological catalogs of galaxies at 0.02<z<0.1
• Navigating AGN variability with self-organizing maps
• Investigation of active galactic nuclei in time domain era