Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?

Abstract

Three important properties of a classification machinery are: (i) the system preserves the core information of the input data; (ii) the training examples convey information about unseen data; and (iii) the system is able to treat differently points from different classes. In this work we show that these fundamental properties are satisfied by the architecture of deep neural networks. We formally prove that these networks with random Gaussian weights perform a distance-preserving embedding of the data, with a special treatment for in-class and out-of-class data. Similar points at the input of the network are likely to have a similar output. The theoretical analysis of deep networks here presented exploits tools used in the compressed sensing and dictionary learning literature, thereby making a formal connection between these important topics. The derived results allow drawing conclusions on the metric learning properties of the network and their relation to its structure, as well as providing bounds on the required size of the training set such that the training examples would represent faithfully the unseen data. The results are validated with state-of-the-art trained networks.

14 pages, 13 figures

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Convolutional Kernel Networks

Cited by in corpus (54)

• Robust Large Margin Deep Neural Networks
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Deep Learning on Graphs: A Survey
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Entropy and mutual information in models of deep neural networks
• A New Learning Paradigm for Random Vector Functional-Link Network: RVFL+
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Change Detection in Heterogeneous Optical and SAR Remote Sensing Images via Deep Homogeneous Feature Fusion
• Deep Divergence-Based Approach to Clustering
• Mathematics of Deep Learning
• Measuring the Intrinsic Dimension of Objective Landscapes
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Constructing coarse-scale bifurcation diagrams from spatio-temporal observations of microscopic simulations: A parsimonious machine learning approach
• How to Start Training: The Effect of Initialization and Architecture
• Change Detection in Graph Streams by Learning Graph Embeddings on Constant-Curvature Manifolds
• Parsimonious Physics-Informed Random Projection Neural Networks for Initial-Value Problems of ODEs and index-1 DAEs
• Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
• Discovering and Deciphering Relationships Across Disparate Data Modalities
• Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey
• Binary embeddings with structured hashed projections
• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• Efficient Representation of Low-Dimensional Manifolds using Deep Networks
• Learning Deep Analysis Dictionaries for Image Super-Resolution
• Measuring and regularizing networks in function space
• Dataset Condensation with Distribution Matching
• Eigenvalue distribution of nonlinear models of random matrices
• Efficient and Private Federated Learning with Partially Trainable Networks
• On the Stability of Deep Networks
• Predictive Analysis of COVID-19 Time-series Data from Johns Hopkins University
• Sketching for Large-Scale Learning of Mixture Models
• How Powerful are Shallow Neural Networks with Bandlimited Random Weights?
• Deep ReLU Networks Preserve Expected Length
• Deep Stochastic Configuration Networks with Universal Approximation Property
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Instance Optimal Decoding and the Restricted Isometry Property
• Exploring the Interchangeability of CNN Embedding Spaces
• Abstracting Deep Neural Networks into Concept Graphs for Concept Level Interpretability
• Johnson-Lindenstrauss Lemma, Linear and Nonlinear Random Projections, Random Fourier Features, and Random Kitchen Sinks: Tutorial and Survey
• High-dimensional Neural Feature Design for Layer-wise Reduction of Training Cost
• Causality-inspired Single-source Domain Generalization for Medical Image Segmentation
• Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case
• Deep Random Projection Outlyingness for Unsupervised Anomaly Detection
• A Conceptual Framework for Lifelong Learning
• Nonasymptotic Guarantees for Spiked Matrix Recovery with Generative Priors
• What training reveals about neural network complexity
• Artistic Domain Generalisation Methods are Limited by their Deep Representations
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• R3Net: Random Weights, Rectifier Linear Units and Robustness for Artificial Neural Network
• Numerical Solution of Stiff ODEs with Physics-Informed RPNNs
• Reservoir Transformers
• The Separation Capacity of Random Neural Networks
• Effective Non-Random Extreme Learning Machine
• On the security relevance of weights in deep learning
• Fast nonlinear embeddings via structured matrices