Understanding Deep Convolutional Networks

Abstract

Deep convolutional networks provide state of the art classifications and regressions results over many high-dimensional problems. We review their architecture, which scatters data with a cascade of linear filter weights and non-linearities. A mathematical framework is introduced to analyze their properties. Computations of invariants involve multiscale contractions, the linearization of hierarchical symmetries, and sparse separations. Applications are discussed.

17 pages, 4 Figures

References in corpus (4)

• Sequence to Sequence Learning with Neural Networks
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• The Loss Surfaces of Multilayer Networks
• Joint Time-Frequency Scattering for Audio Classification

Cited by in corpus (151)

• Geometric deep learning: going beyond Euclidean data
• Deep learning for universal linear embeddings of nonlinear dynamics
• A Review of Convolutional Neural Networks for Inverse Problems in Imaging
• Machine learning in acoustics: theory and applications
• Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
• Adversarial Perturbations Against Deep Neural Networks for Malware Classification
• What makes ImageNet good for transfer learning?
• Biometric Face Presentation Attack Detection with Multi-Channel Convolutional Neural Network
• Compact Convolutional Neural Networks for Classification of Asynchronous Steady-state Visual Evoked Potentials
• An empirical study on evaluation metrics of generative adversarial networks
• Learning to Predict the Cosmological Structure Formation
• Understanding and mitigating gradient pathologies in physics-informed neural networks
• Deep learning of dynamics and signal-noise decomposition with time-stepping constraints
• Robust flow field reconstruction from limited measurements via sparse representation
• Data-driven rogue waves and parameter discovery in the defocusing NLS equation with a potential using the PINN deep learning
• A deep convolutional neural network approach to single-particle recognition in cryo-electron microscopy
• The Modern Mathematics of Deep Learning
• Likelihood-free inference with neural compression of DES SV weak lensing map statistics
• Understanding Convolutional Neural Networks
• Interpretable Convolutional Filters with SincNet
• Deep Convolutional Neural Networks as strong gravitational lens detectors
• Ensemble of Deep Convolutional Neural Networks for Learning to Detect Retinal Vessels in Fundus Images
• Mathematics of Deep Learning
• Scattering Networks for Hybrid Representation Learning
• Joint Time-Frequency Scattering
• Physics-informed machine learning for building performance simulation-A review of a nascent field
• Deep learning dark matter map reconstructions from DES SV weak lensing data
• Deep Learning for Passive Synthetic Aperture Radar
• Using Convolutional Neural Networks to identify Gravitational Lenses in Astronomical images
• Excessive Invariance Causes Adversarial Vulnerability
• i-RevNet: Deep Invertible Networks
• Deep Learning for space-variant deconvolution in galaxy surveys
• A neural network multigrid solver for the Navier-Stokes equations
• Characterizing the Decision Boundary of Deep Neural Networks
• Analysis of time-frequency scattering transforms
• Deep Learning Models for Global Coordinate Transformations that Linearize PDEs
• Biomedical Image Reconstruction: From the Foundations to Deep Neural Networks
• Impacts of DEM Type and Resolution on Deep Learning-Based Flood Inundation Mapping
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• A General Theory of Equivariant CNNs on Homogeneous Spaces
• Galaxy Clustering Analysis with SimBIG and the Wavelet Scattering Transform
• Machine learning for complete intersection Calabi-Yau manifolds: a methodological study
• Data-driven emergence of convolutional structure in neural networks
• Approximation spaces of deep neural networks
• Statistical Physics and Representations in Real and Artificial Neural Networks
• Como funciona o Deep Learning
• Learning with invariances in random features and kernel models
• Geometric compression of invariant manifolds in neural nets
• CIFAR10 to Compare Visual Recognition Performance between Deep Neural Networks and Humans
• Explainable Offline-Online Training of Neural Networks for Parameterizations: A 1D Gravity Wave-QBO Testbed in the Small-data Regime
• Provably scale-covariant continuous hierarchical networks based on scale-normalized differential expressions coupled in cascade
• Decimated Framelet System on Graphs and Fast G-Framelet Transforms
• Optical Flow Estimation using a Spatial Pyramid Network
• Convolution, attention and structure embedding
• Physics-informed machine learning for sensor fault detection with flight test data
• PDE-NetGen 1.0: from symbolic PDE representations of physical processes to trainable neural network representations
• Finding Strong Gravitational Lenses Through Self-Attention
• Beyond the storage capacity: data driven satisfiability transition
• Statistical Hypothesis Testing Based on Machine Learning: Large Deviations Analysis
• Universality of Deep Convolutional Neural Networks
• Mean-Field Neural ODEs via Relaxed Optimal Control
• From Dark Matter to Galaxies with Convolutional Neural Networks
• Multiresolution Convolutional Autoencoders
• SSN: Stockwell Scattering Network for SAR Image Change Detection
• Nonlinear Power Method for Computing Eigenvectors of Proximal Operators and Neural Networks
• Wavelet Conditional Renormalization Group
• Generalization in multitask deep neural classifiers: a statistical physics approach
• Generative Models of Multi-channel Data from a Single Example -- Application to Dust Emission
• Large-time asymptotics in deep learning
• A continuous-time analysis of distributed stochastic gradient
• Learning Waveform-Based Acoustic Models using Deep Variational Convolutional Neural Networks
• Real-time Deep Video Deinterlacing
• Analysis on the Nonlinear Dynamics of Deep Neural Networks: Topological Entropy and Chaos
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• A Mathematical Theory of Attention
• Generative networks as inverse problems with Scattering transforms
• Neural Anisotropy Directions
• On the Importance of Consistency in Training Deep Neural Networks
• Error Approximation and Bias Correction in Dynamic Problems using a Recurrent Neural Network/Finite Element Hybrid Model
• A Statistical Investigation of Long Memory in Language and Music
• Analysis of Invariance and Robustness via Invertibility of ReLU-Networks
• A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
• Scattering Networks on the Sphere for Scalable and Rotationally Equivariant Spherical CNNs
• MVC-Net: A Convolutional Neural Network Architecture for Manifold-Valued Images With Applications
• Fatigue monitoring and maneuver identification for vehicle fleets using a virtual sensing approach
• Relative stability toward diffeomorphisms indicates performance in deep nets
• Observer Dependent Lossy Image Compression
• Scattering Transform Based Image Clustering using Projection onto Orthogonal Complement
• Linear Time Complexity Deep Fourier Scattering Network and Extension to Nonlinear Invariants
• Distributed Sequence Memory of Multidimensional Inputs in Recurrent Networks
• Alignment Distances on Systems of Bags
• Covariance properties under natural image transformations for the generalized Gaussian derivative model for visual receptive fields
• A novel approach to graph distinction through GENEOs and permutants
• Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems
• Gabor frames and deep scattering networks in audio processing
• State-of-Charge Estimation of a Li-Ion Battery using Deep Forward Neural Networks
• Automated Polysomnography Analysis for Detection of Non-Apneic and Non-Hypopneic Arousals using Feature Engineering and a Bidirectional LSTM Network
• Improving Machine Hearing on Limited Data Sets
• Machine Learning S-Wave Scattering Phase Shifts Bypassing the Radial Schrödinger Equation
• SurReal: Complex-Valued Learning as Principled Transformations on a Scaling and Rotation Manifold
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Residual Networks as Geodesic Flows of Diffeomorphisms
• Large-scale image analysis using docker sandboxing
• Building Bayesian Neural Networks with Blocks: On Structure, Interpretability and Uncertainty
• The Shape of RemiXXXes to Come: Audio Texture Synthesis with Time-frequency Scattering
• Physics-Constrained Predictive Molecular Latent Space Discovery with Graph Scattering Variational Autoencoder
• ExplainFix: Explainable Spatially Fixed Deep Networks
• Between Homomorphic Signal Processing and Deep Neural Networks: Constructing Deep Algorithms for Polyphonic Music Transcription
• Optimal deep neural networks for sparse recovery via Laplace techniques
• On the effect of pooling on the geometry of representations
• On the Regularity of Attention
• Approximation properties relative to continuous scale space for hybrid discretizations of Gaussian derivative operators
• Building Function Approximators on top of Haar Scattering Networks
• Learning a Lie Algebra from Unlabeled Data Pairs
• Class Subset Selection for Transfer Learning using Submodularity
• R3Net: Random Weights, Rectifier Linear Units and Robustness for Artificial Neural Network
• GENEOnet: A new machine learning paradigm based on Group Equivariant Non-Expansive Operators. An application to protein pocket detection
• Structural Robustness for Deep Learning Architectures
• Convolutional Neural Networks Analyzed via Inverse Problem Theory and Sparse Representations
• Towards the effectiveness of Deep Convolutional Neural Network based Fast Random Forest Classifier
• Overcomplete Frame Thresholding for Acoustic Scene Analysis
• Accurate Tumor Tissue Region Detection with Accelerated Deep Convolutional Neural Networks
• Harmonic Decompositions of Convolutional Networks
• Examining and Mitigating Kernel Saturation in Convolutional Neural Networks using Negative Images
• The role of invariance in spectral complexity-based generalization bounds
• Gradient-based Filter Design for the Dual-tree Wavelet Transform
• On the Learning Dynamics of Two-layer Nonlinear Convolutional Neural Networks
• One or Two Components? The Scattering Transform Answers
• Quantifying Roman WFI Dark Images with the Wavelet Scattering Transform
• Knowledge as Invariance -- History and Perspectives of Knowledge-augmented Machine Learning
• Graphs for deep learning representations
• Deep Haar Scattering Networks in Pattern Recognition: A promising approach
• Scientific Calculator for Designing Trojan Detectors in Neural Networks
• Sparse Multi-Family Deep Scattering Network
• Graphs as Tools to Improve Deep Learning Methods
• Theory of Deep Convolutional Neural Networks II: Spherical Analysis
• Adversarial Attacks with Time-Scale Representations
• The Expressivity and Training of Deep Neural Networks: toward the Edge of Chaos?
• Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions
• Rediscovering Deep Neural Networks Through Finite-State Distributions
• Scattering Features for Multimodal Gait Recognition
• Dynamic Variational Autoencoders for Visual Process Modeling
• Learning flexible representations of stochastic processes on graphs
• Machine listening intelligence
• Deep Autoencoders: From Understanding to Generalization Guarantees
• Towards glass-box CNNs
• Samplets: A new paradigm for data compression
• Parametrizing filters of a CNN with a GAN
• BLADE: Filter Learning for General Purpose Computational Photography
• Interpretable Image Clustering via Diffeomorphism-Aware K-Means
• On the geometric and Riemannian structure of the spaces of group equivariant non-expansive operators