Set Representation Learning with Generalized Sliced-Wasserstein Embeddings
arXiv:2103.03892
Abstract
An increasing number of machine learning tasks deal with learning representations from set-structured data. Solutions to these problems involve the composition of permutation-equivariant modules (e.g., self-attention, or individual processing via feed-forward neural networks) and permutation-invariant modules (e.g., global average pooling, or pooling by multi-head attention). In this paper, we propose a geometrically-interpretable framework for learning representations from set-structured data, which is rooted in the optimal mass transportation problem. In particular, we treat elements of a set as samples from a probability measure and propose an exact Euclidean embedding for Generalized Sliced Wasserstein (GSW) distances to learn from set-structured data effectively. We evaluate our proposed framework on multiple supervised and unsupervised set learning tasks and demonstrate its superiority over state-of-the-art set representation learning approaches.
References in corpus (13)
- Bootstrap your own latent: A new approach to self-supervised Learning
- Contrastive Representation Learning: A Framework and Review
- Exploring Simple Siamese Representation Learning
- DeepJDOT: Deep Joint Distribution Optimal Transport for Unsupervised Domain Adaptation
- Towards Sparse Hierarchical Graph Classifiers
- A Sparse Multi-Scale Algorithm for Dense Optimal Transport
- Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs
- FSPool: Learning Set Representations with Featurewise Sort Pooling
- An efficient linear programming method for Optimal Transportation
- Subspace Robust Wasserstein Distances
- Rep the Set: Neural Networks for Learning Set Representations
- Conditional Set Generation with Transformers
- A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention