On the Limitations of Representing Functions on Sets

Abstract

Recent work on the representation of functions on sets has considered the use of summation in a latent space to enforce permutation invariance. In particular, it has been conjectured that the dimension of this latent space may remain fixed as the cardinality of the sets under consideration increases. However, we demonstrate that the analysis leading to this conjecture requires mappings which are highly discontinuous and argue that this is only of limited practical use. Motivated by this observation, we prove that an implementation of this model via continuous mappings (as provided by e.g. neural networks or Gaussian processes) actually imposes a constraint on the dimensionality of the latent space. Practical universal function representation for set inputs can only be achieved with a latent dimension at least the size of the maximum number of input elements.

arXiv commentPublished at the International Conference on Machine Learning (2019)

Cited by in corpus (18)

• Point Transformer
• SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
• What Can Neural Networks Reason About?
• Accurate Learning of Graph Representations with Graph Multiset Pooling
• Deep Ensembles vs. Committees for Uncertainty Estimation in Neural-Network Force Fields: Comparison and Application to Active Learning
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks
• Set-to-Sequence Methods in Machine Learning: a Review
• Task-Adaptive Neural Network Search with Meta-Contrastive Learning
• End-to-end Recurrent Multi-Object Tracking and Trajectory Prediction with Relational Reasoning
• Learning Prototype-oriented Set Representations for Meta-Learning
• Walk Message Passing Neural Networks and Second-Order Graph Neural Networks
• Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation
• You Never Cluster Alone
• GP-ConvCNP: Better Generalization for Convolutional Conditional Neural Processes on Time Series Data
• Representing Unordered Data Using Complex-Weighted Multiset Automata
• The CAT SET on the MAT: Cross Attention for Set Matching in Bipartite Hypergraphs
• A Functional Perspective on Learning Symmetric Functions with Neural Networks