4 papers
How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNs
Emily Dent, Jared Tanner
The Edge-of-Chaos (EoC) theory developed for the random initialization of deep networks allows more efficient training by both preserving information in the initial outputs of the…
Theory of Minimal Weight Perturbations in Deep Networks and its Applications for Low-Rank Activated Backdoor Attacks
Bethan Evans, Jared Tanner
The minimal norm weight perturbations of DNNs required to achieve a specified change in output are derived and the factors determining its size are discussed. These single-layer ex…
Deep Neural Network Initialization with Sparsity Inducing Activations
Ilan Price, Nicholas Daultry Ball, Samuel C. H. Lam +2
Inducing and leveraging sparse activations during training and inference is a promising avenue for improving the computational efficiency of deep networks, which is increasingly im…
Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processes
Thiziri Nait-Saada, Alireza Naderi, Jared Tanner
The infinitely wide neural network has been proven a useful and manageable mathematical model that enables the understanding of many phenomena appearing in deep learning. One examp…