Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks

Abstract

Many real-world problems, e.g. object detection, have outputs that are naturally expressed as sets of entities. This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors. We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks. Specifically, in our formulation we incorporate the permutation as unobservable variable and estimate its distribution during the learning process using alternating optimization. We demonstrate the validity of this new formulation on two relevant vision problems: object detection, for which our formulation outperforms state-of-the-art detectors such as Faster R-CNN and YOLO, and a complex CAPTCHA test, where we observe that, surprisingly, our set based network acquired the ability of mimicking arithmetics without any rules being coded.

References in corpus (1)

• Joint Learning of Set Cardinality and State Distribution

Cited by in corpus (9)

• Deep Set Prediction Networks
• Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs
• FSPool: Learning Set Representations with Featurewise Sort Pooling
• Conditional Set Generation with Transformers
• Auto-labelling of Markers in Optical Motion Capture by Permutation Learning
• Hierarchical Poset Decoding for Compositional Generalization in Language
• Better Set Representations For Relational Reasoning
• Reversible Action Design for Combinatorial Optimization with Reinforcement Learning
• Learning to Represent and Predict Sets with Deep Neural Networks