Identifying and Exploiting Structures for Reliable Deep Learning

Abstract

Deep learning research has recently witnessed an impressively fast-paced progress in a wide range of tasks including computer vision, natural language processing, and reinforcement learning. The extraordinary performance of these systems often gives the impression that they can be used to revolutionise our lives for the better. However, as recent works point out, these systems suffer from several issues that make them unreliable for use in the real world, including vulnerability to adversarial attacks (Szegedy et al. [248]), tendency to memorise noise (Zhang et al. [292]), being over-confident on incorrect predictions (miscalibration) (Guo et al. [99]), and unsuitability for handling private data (Gilad-Bachrach et al. [88]). In this thesis, we look at each of these issues in detail, investigate their causes, and propose computationally cheap algorithms for mitigating them in practice. To do this, we identify structures in deep neural networks that can be exploited to mitigate the above causes of unreliability of deep learning algorithms.

Final Thesis for DPhil in Computer Science submitted at the University of Oxford

References in corpus (55)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• Community detection in graphs
• Conditional Generative Adversarial Nets
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Shortcut Learning in Deep Neural Networks
• Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Understanding deep learning requires rethinking generalization
• A Learned Representation For Artistic Style
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Adversarial Robustness via Randomized Smoothing
• Compressing Neural Networks with the Hashing Trick
• Fast is better than free: Revisiting adversarial training
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption
• cGANs with Projection Discriminator
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Evaluating Robustness of Neural Networks with Mixed Integer Programming
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Certified Adversarial Robustness with Additive Noise
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Chiron: Privacy-preserving Machine Learning as a Service
• Bitwise Neural Networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• Adversarial Training Can Hurt Generalization
• A Unified View of Piecewise Linear Neural Network Verification
• Improving Neural Language Models with a Continuous Cache
• Evaluating model calibration in classification
• What Do Compressed Deep Neural Networks Forget?
• Calibrating Deep Neural Networks using Focal Loss
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Differential Privacy Has Disparate Impact on Model Accuracy
• Robust Convolutional Neural Networks under Adversarial Noise
• Complexity of Linear Regions in Deep Networks
• Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation
• Language Models with Transformers
• Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
• Towards GAN Benchmarks Which Require Generalization
• Calibration tests in multi-class classification: A unifying framework
• Adversarially Robust Training through Structured Gradient Regularization
• Diverse Ensembles Improve Calibration
• Multiscale sequence modeling with a learned dictionary
• Robustness via Deep Low-Rank Representations
• DeepSecure: Scalable Provably-Secure Deep Learning
• Query complexity of adversarial attacks