Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods

Abstract

Graph Neural Networks (GNNs) have achieved state-of-the-art performance in node classification, regression, and recommendation tasks. GNNs work well when rich and high-quality connections are available. However, their effectiveness is often jeopardized in many real-world graphs in which node degrees have power-law distributions. The extreme case of this situation, where a node may have no neighbors, is called Strict Cold Start (SCS). SCS forces the prediction to rely completely on the node's own features. We propose Cold Brew, a teacher-student distillation approach to address the SCS and noisy-neighbor challenges for GNNs. We also introduce feature contribution ratio (FCR), a metric to quantify the behavior of inductive GNNs to solve SCS. We experimentally show that FCR disentangles the contributions of different graph data components and helps select the best architecture for SCS generalization. We further demonstrate the superior performance of Cold Brew on several public benchmark and proprietary e-commerce datasets, where many nodes have either very few or noisy connections. Our source code is available at https://github.com/amazon-research/gnn-tail-generalization.

arXiv commentPublished as a conference paper in ICLR 2022

References in corpus (28)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• FitNets: Hints for Thin Deep Nets
• Graph Embedding Techniques, Applications, and Performance: A Survey
• Simplifying Graph Convolutional Networks
• Representation Learning on Graphs with Jumping Knowledge Networks
• Towards Deeper Graph Neural Networks
• How Powerful are Graph Neural Networks?
• DeeperGCN: All You Need to Train Deeper GCNs
• Revisiting Graph Neural Networks: All We Have is Low-Pass Filters
• PairNorm: Tackling Oversmoothing in GNNs
• Geom-GCN: Geometric Graph Convolutional Networks
• Combining Label Propagation and Simple Models Out-performs Graph Neural Networks
• Unifying Graph Convolutional Neural Networks and Label Propagation
• On the Bottleneck of Graph Neural Networks and its Practical Implications
• Revisiting Over-smoothing in Deep GCNs
• STAR-GCN: Stacked and Reconstructed Graph Convolutional Networks for Recommender Systems
• Graph-MLP: Node Classification without Message Passing in Graph
• Understanding Regularized Spectral Clustering via Graph Conductance
• Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation
• Non-Local Graph Neural Networks
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks
• Dissecting the Diffusion Process in Linear Graph Convolutional Networks
• Zero-Shot Recommender Systems
• Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective
• On Self-Distilling Graph Neural Network