Building Compact and Robust Deep Neural Networks with Toeplitz Matrices

Abstract

Deep neural networks are state-of-the-art in a wide variety of tasks, however, they exhibit important limitations which hinder their use and deployment in real-world applications. When developing and training neural networks, the accuracy should not be the only concern, neural networks must also be cost-effective and reliable. Although accurate, large neural networks often lack these properties. This thesis focuses on the problem of training neural networks which are not only accurate but also compact, easy to train, reliable and robust to adversarial examples. To tackle these problems, we leverage the properties of structured matrices from the Toeplitz family to build compact and secure neural networks.

arXiv commentThesis

References in corpus (39)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Federated Learning: Strategies for Improving Communication Efficiency
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Wide Residual Networks
• Scaling Laws for Neural Language Models
• Do Deep Nets Really Need to be Deep?
• A guide to convolution arithmetic for deep learning
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Theoretically Principled Trade-off between Robustness and Accuracy
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Certified Adversarial Robustness via Randomized Smoothing
• On Evaluating Adversarial Robustness
• Compressing Neural Networks with the Hashing Trick
• Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Learnable pooling with Context Gating for video classification
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Memory Bounded Deep Convolutional Networks
• Circulant Binary Embedding
• Compressing Recurrent Neural Network with Tensor Train
• Ultimate tensorization: compressing convolutional and FC layers alike
• Plug-and-Play Methods Provably Converge with Properly Trained Denoisers
• A Tensorized Transformer for Language Modeling
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Implicit Regularization in Deep Learning
• Ternary Neural Networks with Fine-Grained Quantization
• Temporal Modeling Approaches for Large-scale Youtube-8M Video Understanding
• Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations
• Compact Nonlinear Maps and Circulant Extensions
• Curse of Dimensionality on Randomized Smoothing for Certifiable Robustness
• Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks
• Lipschitz constant estimation of Neural Networks via sparse polynomial optimization
• Fast Approximation of Rotations and Hessians matrices
• Deep Learning Methods for Efficient Large Scale Video Labeling
• Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
• Bounding Multivariate Trigonometric Polynomials with Applications to Filter Bank Design
• The coupling effect of Lipschitz regularization in deep neural networks