Training Efficiency and Robustness in Deep Learning

Abstract

Deep Learning has revolutionized machine learning and artificial intelligence, achieving superhuman performance in several standard benchmarks. It is well-known that deep learning models are inefficient to train; they learn by processing millions of training data multiple times and require powerful computational resources to process large batches of data in parallel at the same time rather than sequentially. Deep learning models also have unexpected failure modes; they can be fooled into misbehaviour, producing unexpectedly incorrect predictions. In this thesis, we study approaches to improve the training efficiency and robustness of deep learning models. In the context of learning visual-semantic embeddings, we find that prioritizing learning on more informative training data increases convergence speed and improves generalization performance on test data. We formalize a simple trick called hard negative mining as a modification to the learning objective function with no computational overhead. Next, we seek improvements to optimization speed in general-purpose optimization methods in deep learning. We show that a redundancy-aware modification to the sampling of training data improves the training speed and develops an efficient method for detecting the diversity of training signal, namely, gradient clustering. Finally, we study adversarial robustness in deep learning and approaches to achieve maximal adversarial robustness without training with additional data. For linear models, we prove guaranteed maximal robustness achieved only by appropriate choice of the optimizer, regularization, or architecture.

arXiv commentA thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy

References in corpus (68)

• Explaining and Harnessing Adversarial Examples
• A Simple Framework for Contrastive Learning of Visual Representations
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Intriguing properties of neural networks
• Learning Transferable Visual Models From Natural Language Supervision
• Improved Baselines with Momentum Contrastive Learning
• A Survey of Deep Learning Techniques for Autonomous Driving
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Zero-Shot Text-to-Image Generation
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Theoretically Principled Trade-off between Robustness and Accuracy
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Adversarial Robustness via Randomized Smoothing
• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Robustness May Be at Odds with Accuracy
• Revisiting Small Batch Training for Deep Neural Networks
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Robust Large Margin Deep Neural Networks
• The generalization error of random features regression: Precise asymptotics and double descent curve
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Adversarial Robustness through Local Linearization
• On Empirical Comparisons of Optimizers for Deep Learning
• Measuring the Effects of Data Parallelism on Neural Network Training
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks
• RobustBench: a standardized adversarial robustness benchmark
• Gradient descent aligns the layers of deep linear networks
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Smooth Adversarial Training
• Direct Optimization of Ranking Measures
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Low Frequency Adversarial Perturbation
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Variance Reduction in SGD by Distributed Importance Sampling
• Accelerating Minibatch Stochastic Gradient Descent using Stratified Sampling
• First-order Adversarial Vulnerability of Neural Networks and Input Dimension
• Batch Normalization is a Cause of Adversarial Vulnerability
• Biased Importance Sampling for Deep Neural Network Training
• Understanding and Mitigating the Tradeoff Between Robustness and Accuracy
• Active Contrastive Learning of Audio-Visual Video Representations
• Are All Training Examples Created Equal? An Empirical Study
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
• On the Noisy Gradient Descent that Generalizes as SGD
• Accelerating Deep Learning by Focusing on the Biggest Losers
• Efficient Per-Example Gradient Computations
• Scalable Second Order Optimization for Deep Learning
• Beyond Pixel Norm-Balls: Parametric Adversaries using an Analytically Differentiable Renderer
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Margins, Shrinkage, and Boosting
• An Effective Anti-Aliasing Approach for Residual Networks
• Towards an Adversarially Robust Normalization Approach
• Should Adversarial Attacks Use Pixel p-Norm?
• A Study of Gradient Variance in Deep Learning
• A Unifying View on Implicit Bias in Training Linear Neural Networks
• NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
• When Does Preconditioning Help or Hurt Generalization?
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
• Increasing-Margin Adversarial (IMA) Training to Improve Adversarial Robustness of Neural Networks
• To Each Optimizer a Norm, To Each Norm its Generalization