Generalization bounds via distillation

Abstract

This paper theoretically investigates the following empirical phenomenon: given a high-complexity network with poor generalization bounds, one can distill it into a network with nearly identical predictions but low complexity and vastly smaller generalization bounds. The main contribution is an analysis showing that the original network inherits this good generalization bound from its distillation, assuming the use of well-behaved data augmentation. This bound is presented both in an abstract and in a concrete form, the latter complemented by a reduction technique to handle modern computation graphs featuring convolutional layers, fully-connected layers, and skip connections, to name a few. To round out the story, a (looser) classical uniform convergence analysis of compression is also presented, as well as a variety of experiments on cifar and mnist demonstrating similar generalization performance between the original network and its distillation.

To appear, ICLR 2021

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Understanding deep learning requires rethinking generalization
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
• Uniform convergence may be unable to explain generalization in deep learning
• Optimistic Rates for Learning with a Smooth Loss
• Analysis of a randomized approximation scheme for matrix multiplication

Cited by in corpus (8)

• Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms
• Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks
• Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learning
• Self-training Converts Weak Learners to Strong Learners in Mixture Models
• On Predicting Generalization using GANs
• Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD
• Compression Implies Generalization