Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation

Abstract

Most set prediction models in deep learning use set-equivariant operations, but they actually operate on multisets. We show that set-equivariant functions cannot represent certain functions on multisets, so we introduce the more appropriate notion of multiset-equivariance. We identify that the existing Deep Set Prediction Network (DSPN) can be multiset-equivariant without being hindered by set-equivariance and improve it with approximate implicit differentiation, allowing for better optimization while being faster and saving memory. In a range of toy experiments, we show that the perspective of multiset-equivariance is beneficial and that our changes to DSPN achieve better results in most cases. On CLEVR object property prediction, we substantially improve over the state-of-the-art Slot Attention from 8% to 77% in one of the strictest evaluation metrics because of the benefits made possible by implicit differentiation.

arXiv commentPublished at International Conference on Learning Representations (ICLR) 2022

References in corpus (15)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Learning Representations and Generative Models for 3D Point Clouds
• Learning Deep Generative Models of Graphs
• Deep Equilibrium Models
• Multi-Object Representation Learning with Iterative Variational Inference
• Meta-Learning with Implicit Gradients
• Differentiable Convex Optimization Layers
• On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization
• Deep Set Prediction Networks
• Implicit Graph Neural Networks
• FSPool: Learning Set Representations with Featurewise Sort Pooling
• Efficient and Modular Implicit Differentiation
• On the Limitations of Representing Functions on Sets
• Conditional Set Generation with Transformers
• iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients