Learning to Represent and Predict Sets with Deep Neural Networks

Abstract

In this thesis, we develop various techniques for working with sets in machine learning. Each input or output is not an image or a sequence, but a set: an unordered collection of multiple objects, each object described by a feature vector. Their unordered nature makes them suitable for modeling a wide variety of data, ranging from objects in images to point clouds to graphs. Deep learning has recently shown great success on other types of structured data, so we aim to build the necessary structures for sets into deep neural networks. The first focus of this thesis is the learning of better set representations (sets as input). Existing approaches have bottlenecks that prevent them from properly modeling relations between objects within the set. To address this issue, we develop a variety of techniques for different scenarios and show that alleviating the bottleneck leads to consistent improvements across many experiments. The second focus of this thesis is the prediction of sets (sets as output). Current approaches do not take the unordered nature of sets into account properly. We determine that this results in a problem that causes discontinuity issues with many set prediction tasks and prevents them from learning some extremely simple datasets. To avoid this problem, we develop two models that properly take the structure of sets into account. Various experiments show that our set prediction techniques can significantly benefit over existing approaches.

arXiv commentPhD thesis submitted December 2019

References in corpus (17)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Recurrent Models of Visual Attention
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Learning Representations and Generative Models for 3D Point Clouds
• A simple neural network module for relational reasoning
• GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models
• MolGAN: An implicit generative model for small molecular graphs
• Learning Deep Generative Models of Graphs
• Multi-Object Representation Learning with Iterative Variational Inference
• Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
• Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
• Ranking via Sinkhorn Propagation
• Bilinear Attention Networks
• Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs
• Interpretable Counting for Visual Question Answering
• Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks
• Loss Functions for Multiset Prediction