Training Efficiency and Robustness in Deep Learning

Abstract

Deep Learning has revolutionized machine learning and artificial intelligence, achieving superhuman performance in several standard benchmarks. It is well-known that deep learning models are inefficient to train; they learn by processing millions of training data multiple times and require powerful computational resources to process large batches of data in parallel at the same time rather than sequentially. Deep learning models also have unexpected failure modes; they can be fooled into misbehaviour, producing unexpectedly incorrect predictions. In this thesis, we study approaches to improve the training efficiency and robustness of deep learning models. In the context of learning visual-semantic embeddings, we find that prioritizing learning on more informative training data increases convergence speed and improves generalization performance on test data. We formalize a simple trick called hard negative mining as a modification to the learning objective function with no computational overhead. Next, we seek improvements to optimization speed in general-purpose optimization methods in deep learning. We show that a redundancy-aware modification to the sampling of training data improves the training speed and develops an efficient method for detecting the diversity of training signal, namely, gradient clustering. Finally, we study adversarial robustness in deep learning and approaches to achieve maximal adversarial robustness without training with additional data. For linear models, we prove guaranteed maximal robustness achieved only by appropriate choice of the optimizer, regularization, or architecture.

A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy

References in corpus (20)

• Explaining and Harnessing Adversarial Examples
• Learning Transferable Visual Models From Natural Language Supervision
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Zero-Shot Text-to-Image Generation
• Theoretically Principled Trade-off between Robustness and Accuracy
• Certified Adversarial Robustness via Randomized Smoothing
• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Direct Optimization of Ranking Measures
• Accelerating Minibatch Stochastic Gradient Descent using Stratified Sampling
• Batch Normalization is a Cause of Adversarial Vulnerability
• Biased Importance Sampling for Deep Neural Network Training
• Accelerating Deep Learning by Focusing on the Biggest Losers
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Margins, Shrinkage, and Boosting
• An Effective Anti-Aliasing Approach for Residual Networks
• Towards an Adversarially Robust Normalization Approach
• Should Adversarial Attacks Use Pixel p-Norm?
• A Study of Gradient Variance in Deep Learning