Extracting the main trend in a dataset: the Sequencer algorithm

Abstract

Scientists aim to extract simplicity from observations of the complex world. An important component of this process is the exploration of data in search of trends. In practice, however, this tends to be more of an art than a science. Among all trends existing in the natural world, one-dimensional trends, often called sequences, are of particular interest as they provide insights into simple phenomena. However, some are challenging to detect as they may be expressed in complex manners. We present the Sequencer, an algorithm designed to generically identify the main trend in a dataset. It does so by constructing graphs describing the similarities between pairs of observations, computed with a set of metrics and scales. Using the fact that continuous trends lead to more elongated graphs, the algorithm can identify which aspects of the data are relevant in establishing a global sequence. Such an approach can be used beyond the proposed algorithm and can optimize the parameters of any dimensionality reduction technique. We demonstrate the power of the Sequencer using real-world data from astronomy, geology as well as images from the natural world. We show that, in a number of cases, it outperforms the popular t-SNE and UMAP dimensionality reduction techniques. This approach to exploratory data analysis, which does not rely on training nor tuning of any parameter, has the potential to enable discoveries in a wide range of scientific domains. The source code is available on github and we provide an online interface at \url{http://sequencer.org}.

arXiv commentCode available at: https://github.com/dalya/Sequencer. Comments are welcome!

References in corpus (8)

• The Sloan Digital Sky Survey: Technical Summary
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• A Catalog of Broad Absorption Line Quasars from the Sloan Digital Sky Survey Third Data Release
• A Catalog of Broad Absorption Line Quasars in Sloan Digital Sky Survey Data Release 5
• Machine learning meets network science: dimensionality reduction for fast and efficient embedding of networks in the hyperbolic space
• Machine Learning in Astronomy: a practical overview
• Sequencing seismograms: A panoptic view of scattering in the core-mantle boundary region
• Black hole mass estimation for Active Galactic Nuclei from a new angle

Cited by in corpus (6)

• The DAWES review 10: The impact of deep learning for the analysis of galaxy surveys
• Capturing the physics of MaNGA galaxies with self-supervised Machine Learning
• Topology of Pulsar Profiles (ToPP). I. Graph theory method and classification of the EPN
• Characterization of Supernovae Based on the Spectral-Temporal Energy Distribution: Possible two SN Ib Subtypes
• Millisecond pulsars phenomenology under the light of graph theory
• The Galaxy Activity, Torus, and Outflow Survey (GATOS): N. Unveiling physical processes in local active galaxies. Unsupervised hierarchical clustering of JWST MIRI/MRS observations