Deep Multimodal Subspace Clustering Networks

Abstract

We present convolutional neural network (CNN) based approaches for unsupervised multimodal subspace clustering. The proposed framework consists of three main stages - multimodal encoder, self-expressive layer, and multimodal decoder. The encoder takes multimodal data as input and fuses them to a latent space representation. The self-expressive layer is responsible for enforcing the self-expressiveness property and acquiring an affinity matrix corresponding to the data points. The decoder reconstructs the original input data. The network uses the distance between the decoder's reconstruction and the original input in its training. We investigate early, late and intermediate fusion techniques and propose three different encoders corresponding to them for spatial fusion. The self-expressive layers and multimodal decoders are essentially the same for different spatial fusion-based approaches. In addition to various spatial fusion-based methods, an affinity fusion-based network is also proposed in which the self-expressive layer corresponding to different modalities is enforced to be the same. Extensive experiments on three datasets show that the proposed methods significantly outperform the state-of-the-art multimodal subspace clustering methods.

References in corpus (2)

• Natural Scales in Geographical Patterns
• Efficient Large-Scale Multi-Modal Classification

Cited by in corpus (13)

• Deep Embedded Multi-view Clustering with Collaborative Training
• Pseudo-supervised Deep Subspace Clustering
• Self-Supervised Information Bottleneck for Deep Multi-View Subspace Clustering
• A Clustering-guided Contrastive Fusion for Multi-view Representation Learning
• Deep Sparse Representation-based Classification
• MORI-RAN: Multi-view Robust Representation Learning via Hybrid Contrastive Fusion
• A Critique of Self-Expressive Deep Subspace Clustering
• Deep Multi-view Learning to Rank
• Beyond Linear Subspace Clustering: A Comparative Study of Nonlinear Manifold Clustering Algorithms
• Multi-view Subspace Clustering Networks with Local and Global Graph Information
• Variational Inference for Deep Probabilistic Canonical Correlation Analysis
• Robust Self-Supervised Convolutional Neural Network for Subspace Clustering and Classification
• fMBN-E: Efficient Unsupervised Network Structure Ensemble and Selection for Clustering