SpectralNet: Spectral Clustering using Deep Neural Networks

Abstract

Spectral clustering is a leading and popular technique in unsupervised data analysis. Two of its major limitations are scalability and generalization of the spectral embedding (i.e., out-of-sample-extension). In this paper we introduce a deep learning approach to spectral clustering that overcomes the above shortcomings. Our network, which we call SpectralNet, learns a map that embeds input data points into the eigenspace of their associated graph Laplacian matrix and subsequently clusters them. We train SpectralNet using a procedure that involves constrained stochastic optimization. Stochastic optimization allows it to scale to large datasets, while the constraints, which are implemented using a special-purpose output layer, allow us to keep the network output orthogonal. Moreover, the map learned by SpectralNet naturally generalizes the spectral embedding to unseen data points. To further improve the quality of the clustering, we replace the standard pairwise Gaussian affinities with affinities leaned from unlabeled data using a Siamese network. Additional improvement can be achieved by applying the network to code representations produced, e.g., by standard autoencoders. Our end-to-end learning procedure is fully unsupervised. In addition, we apply VC dimension theory to derive a lower bound on the size of SpectralNet. State-of-the-art clustering results are reported on the Reuters dataset. Our implementation is publicly available at https://github.com/kstant0725/SpectralNet .

arXiv commentAdded citations. Accepted to ICLR 2018

References in corpus (7)

• Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
• Deep Learning with Nonparametric Clustering
• Deep Clustering via Joint Convolutional Autoencoder Embedding and Relative Entropy Minimization
• SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation

Cited by in corpus (35)

• Examining COVID-19 Forecasting using Spatio-Temporal Graph Neural Networks
• Spectral Clustering with Graph Neural Networks for Graph Pooling
• Revisiting Over-smoothing in Deep GCNs
• AutoEmbedder: A semi-supervised DNN embedding system for clustering
• Refining a $k$-nearest neighbor graph for a computationally efficient spectral clustering
• Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation
• GAP: Generalizable Approximate Graph Partitioning Framework
• Self-supervised Representation Learning With Path Integral Clustering For Speaker Diarization
• Spectral Feature Transformation for Person Re-identification
• A Framework for Deep Constrained Clustering -- Algorithms and Advances
• Deep Image Clustering with Category-Style Representation
• Supporting Clustering with Contrastive Learning
• Improving ClusterGAN Using Self-Augmented Information Maximization of Disentangling Latent Spaces
• Mask-GVAE: Blind Denoising Graphs via Partition
• The Laplacian in RL: Learning Representations with Efficient Approximations
• Consistent Representation Learning for High Dimensional Data Analysis
• Enhancing ensemble learning and transfer learning in multimodal data analysis by adaptive dimensionality reduction
• Adaptive Graph Auto-Encoder for General Data Clustering
• Tree-SNE: Hierarchical Clustering and Visualization Using t-SNE
• Deep Contrastive Graph Representation via Adaptive Homotopy Learning
• Interpretable Visualizations with Differentiating Embedding Networks
• Convergence of Graph Laplacian with kNN Self-tuned Kernels
• Cluster Analysis with Deep Embeddings and Contrastive Learning
• Modes of Homogeneous Gradient Flows
• Dissimilarity Mixture Autoencoder for Deep Clustering
• Variational Diffusion Autoencoders with Random Walk Sampling
• Double cycle-consistent generative adversarial network for unsupervised conditional generation
• Deep Transformation-Invariant Clustering
• Neural Bayes: A Generic Parameterization Method for Unsupervised Representation Learning
• Multi-level Feature Learning on Embedding Layer of Convolutional Autoencoders and Deep Inverse Feature Learning for Image Clustering
• Spectral Clustering with Smooth Tiny Clusters
• SA-Net: A deep spectral analysis network for image clustering
• Deep Goal-Oriented Clustering
• Adversarial confidence and smoothness regularizations for scalable unsupervised discriminative learning
• Regularized Non-negative Spectral Embedding for Clustering