Deep Models of Interactions Across Sets

Abstract

We use deep learning to model interactions across two or more sets of objects, such as user-movie ratings, protein-drug bindings, or ternary user-item-tag interactions. The canonical representation of such interactions is a matrix (or a higher-dimensional tensor) with an exchangeability property: the encoding's meaning is not changed by permuting rows or columns. We argue that models should hence be Permutation Equivariant (PE): constrained to make the same predictions across such permutations. We present a parameter-sharing scheme and prove that it could not be made any more expressive without violating PE. This scheme yields three benefits. First, we demonstrate state-of-the-art performance on multiple matrix completion benchmarks. Second, our models require a number of parameters independent of the numbers of objects, and thus scale well to large datasets. Third, models can be queried about new objects that were not available at training time, but for which interactions have since been observed. In experiments, our models achieved surprisingly good generalization performance on this matrix extrapolation task, both within domains (e.g., new users and new movies drawn from the same distribution used for training) and even across domains (e.g., predicting music ratings after training on movies).

References in corpus (8)

• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Geometric deep learning: going beyond Euclidean data
• Graph Convolutional Matrix Completion
• Revisiting Semi-Supervised Learning with Graph Embeddings
• Matrix Completion on Graphs
• Neural Network Matrix Factorization
• Equivariance Through Parameter-Sharing

Cited by in corpus (29)

• Universal Invariant and Equivariant Graph Neural Networks
• A Survey on The Expressive Power of Graph Neural Networks
• Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs
• On Learning Sets of Symmetric Elements
• STAR-GCN: Stacked and Reconstructed Graph Convolutional Networks for Recommender Systems
• Universal approximations of permutation invariant/equivariant functions by deep neural networks
• Relational Pooling for Graph Representations
• Natural Graph Networks
• Inductive Matrix Completion Based on Graph Neural Networks
• INMO: A Model-Agnostic and Scalable Module for Inductive Collaborative Filtering
• DOLORES: Deep Contextualized Knowledge Graph Embeddings
• On the Universality of Invariant Networks
• Universal Equivariant Multilayer Perceptrons
• Deep Hyperedges: a Framework for Transductive and Inductive Learning on Hypergraphs
• Using Interpretable Machine Learning to Massively Increase the Number of Antibody-Virus Interactions Across Studies
• Solving Continual Combinatorial Selection via Deep Reinforcement Learning
• Regularizing Towards Permutation Invariance in Recurrent Models
• Equivariant Maps for Hierarchical Structures
• Item Graph Convolution Collaborative Filtering for Inductive Recommendations
• Equivariant Entity-Relationship Networks
• A Computationally Efficient Neural Network Invariant to the Action of Symmetry Subgroups
• Learning Robust Algorithms for Online Allocation Problems Using Adversarial Training
• Mining Entity Synonyms with Efficient Neural Set Generation
• Supervised Whole DAG Causal Discovery
• Adversarial Monte Carlo Meta-Learning of Optimal Prediction Procedures
• Relational dynamic memory networks
• Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views?
• Representation Learning with Multisets
• Distribution-Based Invariant Deep Networks for Learning Meta-Features