Bi-cross validation for estimating spectral clustering hyper parameters

Abstract

One challenge impeding the analysis of terabyte scale x-ray scattering data from the Linac Coherent Light Source LCLS, is determining the number of clusters required for the execution of traditional clustering algorithms. Here we demonstrate that previous work using bi-cross validation (BCV) to determine the number of singular vectors directly maps to the spectral clustering problem of estimating both the number of clusters and hyper parameter values. These results indicate that the process of estimating the number of clusters should not be divorced from the process of estimating other hyper parameters. Applying this method to LCLS x-ray scattering data enables the identification of dropped shots without manually setting boundaries on detector fluence and provides a path towards identifying rare and anomalous events.

5 pages, 4 figures

References in corpus (7)

• Scikit-learn: Machine Learning in Python
• The NumPy array: a structure for efficient numerical computation
• How to generate random matrices from the classical compact groups
• Bi-cross-validation of the SVD and the nonnegative matrix factorization
• Ultrafast X-Ray Induced Changes of the Electronic and Magnetic Response of Solids Due to Valence Electron Redistribution
• Cross-Validation for Unsupervised Learning
• Estimating the number of clusters using cross-validation