Learning Robust Representations via Multi-View Information Bottleneck

Abstract

The information bottleneck principle provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label while minimizing the amount of other, excess information in the representation. The original formulation, however, requires labeled data to identify the superfluous information. In this work, we extend this ability to the multi-view unsupervised setting, where two views of the same underlying entity are provided but the label is unknown. This enables us to identify superfluous information as that not shared by both views. A theoretical analysis leads to the definition of a new multi-view model that produces state-of-the-art results on the Sketchy dataset and label-limited versions of the MIR-Flickr dataset. We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to common unsupervised approaches for representation learning.

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• The information bottleneck method
• Learning deep representations by mutual information estimation and maximization
• Variational Inference with Normalizing Flows
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• MINE: Mutual Information Neural Estimation
• On Mutual Information Maximization for Representation Learning
• Understanding disentangling in $\beta$-VAE
• Variational Lossy Autoencoder
• Deep Variational Canonical Correlation Analysis
• PixelVAE: A Latent Variable Model for Natural Images
• Sketch-a-Net that Beats Humans
• Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval

Cited by in corpus (13)

• Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations
• Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representations
• From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
• Cross-Domain Few-Shot Learning by Representation Fusion
• Multi-Head Self-Attention via Vision Transformer for Zero-Shot Learning
• Multi-View Class Incremental Learning
• Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification
• Unsupervised Hashing with Contrastive Information Bottleneck
• Multi-Task Variational Information Bottleneck
• Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
• Learning Robust Variational Information Bottleneck with Reference
• The Impact of Negative Sampling on Contrastive Structured World Models
• Multi-view Alignment and Generation in CCA via Consistent Latent Encoding