Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration

Abstract

The hypothesis that sub-network initializations (lottery) exist within the initializations of over-parameterized networks, which when trained in isolation produce highly generalizable models, has led to crucial insights into network initialization and has enabled efficient inferencing. Supervised models with uncalibrated confidences tend to be overconfident even when making wrong prediction. In this paper, for the first time, we study how explicit confidence calibration in the over-parameterized network impacts the quality of the resulting lottery tickets. More specifically, we incorporate a suite of calibration strategies, ranging from mixup regularization, variance-weighted confidence calibration to the newly proposed likelihood-based calibration and normalized bin assignment strategies. Furthermore, we explore different combinations of architectures and datasets, and make a number of key findings about the role of confidence calibration. Our empirical studies reveal that including calibration mechanisms consistently lead to more effective lottery tickets, in terms of accuracy as well as empirical calibration metrics, even when retrained using data with challenging distribution shifts with respect to the source dataset.

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• mixup: Beyond Empirical Risk Minimization
• On Calibration of Modern Neural Networks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Variational Dropout Sparsifies Deep Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
• Measuring Calibration in Deep Learning
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Rigging the Lottery: Making All Tickets Winners
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Sparse Transfer Learning via Winning Lottery Tickets

Cited by in corpus (2)

• Towards Domain Generalization for ECG and EEG Classification: Algorithms and Benchmarks
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness