Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs

Abstract

We consider a simple and overarching representation for permutation-invariant functions of sequences (or multiset functions). Our approach, which we call Janossy pooling, expresses a permutation-invariant function as the average of a permutation-sensitive function applied to all reorderings of the input sequence. This allows us to leverage the rich and mature literature on permutation-sensitive functions to construct novel and flexible permutation-invariant functions. If carried out naively, Janossy pooling can be computationally prohibitive. To allow computational tractability, we consider three kinds of approximations: canonical orderings of sequences, functions with $k$-order interactions, and stochastic optimization algorithms with random permutations. Our framework unifies a variety of existing work in the literature, and suggests possible modeling and algorithmic extensions. We explore a few in our experiments, which demonstrate improved performance over current state-of-the-art methods.

arXiv commentThis version clarifies and adds detail to some of the arguments

References in corpus (13)

• Semi-Supervised Classification with Graph Convolutional Networks
• Neural Message Passing for Quantum Chemistry
• Graph Convolutional Matrix Completion
• Representation Learning on Graphs with Jumping Knowledge Networks
• Hierarchical Graph Representation Learning with Differentiable Pooling
• Predicting multicellular function through multi-layer tissue networks
• Deep Models of Interactions Across Sets
• Probabilistic symmetries and invariant neural networks
• GeniePath: Graph Neural Networks with Adaptive Receptive Paths
• Deep Learning with Sets and Point Clouds
• Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks
• BRUNO: A Deep Recurrent Model for Exchangeable Data
• Interpretable Set Functions

Cited by in corpus (28)

• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• How Powerful are Graph Neural Networks?
• Multipole Graph Neural Operator for Parametric Partial Differential Equations
• Neural Operator: Graph Kernel Network for Partial Differential Equations
• On Learning Sets of Symmetric Elements
• Likelihood-Free Parameter Estimation with Neural Bayes Estimators
• Equivariant Energy Flow Networks for Jet Tagging
• Subgraph Neural Networks
• Pointer Graph Networks
• NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphs
• Zero-Shot Learning with Common Sense Knowledge Graphs
• Frame Averaging for Invariant and Equivariant Network Design
• MetaFun: Meta-Learning with Iterative Functional Updates
• FOBE and HOBE: First- and High-Order Bipartite Embeddings
• Are Graph Neural Networks Miscalibrated?
• Regularizing Towards Permutation Invariance in Recurrent Models
• Equivariant Maps for Hierarchical Structures
• Set-to-Sequence Methods in Machine Learning: a Review
• Set Representation Learning with Generalized Sliced-Wasserstein Embeddings
• Breaking the Expressive Bottlenecks of Graph Neural Networks
• Adaptive Neural Message Passing for Inductive Learning on Hypergraphs
• Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding
• Set Aggregation Network as a Trainable Pooling Layer
• Unsupervised Hierarchical Graph Representation Learning by Mutual Information Maximization
• PiNet: Attention Pooling for Graph Classification
• META${}^2$: Memory-efficient taxonomic classification and abundance estimation for metagenomics with deep learning
• Representing Unordered Data Using Complex-Weighted Multiset Automata
• Learning to Represent and Predict Sets with Deep Neural Networks