Information-theoretic generalization bounds for black-box learning algorithms

Abstract

We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm. These bounds improve over the existing information-theoretic bounds, are applicable to a wider range of algorithms, and solve two key challenges: (a) they give meaningful results for deterministic algorithms and (b) they are significantly easier to estimate. We show experimentally that the proposed bounds closely follow the generalization gap in practical scenarios for deep learning.

arXiv commentNeurIPS 2021

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• Understanding deep learning requires rethinking generalization
• Learners that Use Little Information
• Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates
• Information-Theoretic Generalization Bounds for Stochastic Gradient Descent
• Membership Inference Attacks on Deep Regression Models for Neuroimaging

Cited by in corpus (3)

• Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learning
• Formal limitations of sample-wise information-theoretic generalization bounds
• Comparing Comparators in Generalization Bounds