Ensemble Robustness and Generalization of Stochastic Deep Learning Algorithms

Abstract

The question why deep learning algorithms generalize so well has attracted increasing research interest. However, most of the well-established approaches, such as hypothesis capacity, stability or sparseness, have not provided complete explanations (Zhang et al., 2016; Kawaguchi et al., 2017). In this work, we focus on the robustness approach (Xu & Mannor, 2012), i.e., if the error of a hypothesis will not change much due to perturbations of its training examples, then it will also generalize well. As most deep learning algorithms are stochastic (e.g., Stochastic Gradient Descent, Dropout, and Bayes-by-backprop), we revisit the robustness arguments of Xu & Mannor, and introduce a new approach, ensemble robustness, that concerns the robustness of a population of hypotheses. Through the lens of ensemble robustness, we reveal that a stochastic learning algorithm can generalize well as long as its sensitiveness to adversarial perturbations is bounded in average over training examples. Moreover, an algorithm may be sensitive to some adversarial examples (Goodfellow et al., 2015) but still generalize well. To support our claims, we provide extensive simulations for different deep learning algorithms and different network architectures exhibiting a strong correlation between ensemble robustness and the ability to generalize.

16 pages, 2 figures

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Understanding deep learning requires rethinking generalization
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
• A Closer Look at Memorization in Deep Networks
• Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networks
• To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout

Cited by in corpus (8)

• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• Support Vector Machine classification of strong gravitational lenses
• Semi-supervised deep learning by metric embedding
• Attack and Defense of Dynamic Analysis-Based, Adversarial Neural Malware Classification Models
• Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimization
• Generalization in Machine Learning via Analytical Learning Theory
• Deep Online Convex Optimization with Gated Games