Are Graph Neural Networks Miscalibrated?
arXiv:1905.02296
Abstract
Graph Neural Networks (GNNs) have proven to be successful in many classification tasks, outperforming previous state-of-the-art methods in terms of accuracy. However, accuracy alone is not enough for high-stakes decision making. Decision makers want to know the likelihood that a specific GNN prediction is correct. For this purpose, obtaining calibrated models is essential. In this work, we perform an empirical evaluation of the calibration of state-of-the-art GNNs on multiple datasets. Our experiments show that GNNs can be calibrated in some datasets but also badly miscalibrated in others, and that state-of-the-art calibration methods are helpful but do not fix the problem.
Presented at the ICML 2019 Workshop on Learning and Reasoning with Graph-Structured Data
References in corpus (14)
- Scikit-learn: Machine Learning in Python
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Neural Message Passing for Quantum Chemistry
- Fast Graph Representation Learning with PyTorch Geometric
- Convolutional Networks on Graphs for Learning Molecular Fingerprints
- Pitfalls of Graph Neural Network Evaluation
- Efficient Algorithms for t-distributed Stochastic Neighborhood Embedding
- Learning Multiagent Communication with Backpropagation
- Graph networks as learnable physics engines for inference and control
- Evaluating model calibration in classification
- Discovering objects and their relations from entangled scene representations
- Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs
- Deep Weighted Averaging Classifiers
- Stochastic Gradient Descent for Relational Logistic Regression via Partial Network Crawls