355 citations · 1.6k across the 54 of their papers we have counts for
33 papers · 1 filter
Norm-based Generalization Bounds for Compositionally Sparse Neural Networks
Tomer Galanti, Mengjia Xu, Liane Galanti +1
In this paper, we investigate the Rademacher complexity of deep sparse neural networks, where each neuron receives a small number of inputs. We prove generalization bounds for mult…
SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network
Tomer Galanti, Zachary S. Siegel, Aparna Gupte +1
We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks. Our results demonstrate…
Distribution of Classification Margins: Are All Data Equal?
Andrzej Banburski, Fernanda De La Torre, Nishka Pant +2
Recent theoretical results show that gradient descent on deep neural networks under exponential loss functions locally maximizes classification margin, which is equivalent to minim…
The Effects of Image Distribution and Task on Adversarial Robustness
Owen Kunhardt, Arturo Deza, Tomaso Poggio
In this paper, we propose an adaptation to the area under the curve (AUC) metric to measure the adversarial robustness of a model over a particular -interval (inter…
Explicit regularization and implicit bias in deep network classifiers trained with the square loss
Tomaso Poggio, Qianli Liao
Deep ReLU networks trained with the square loss have been observed to perform well in classification tasks. We provide here a theoretical justification based on analysis of the ass…
Biologically Inspired Mechanisms for Adversarial Robustness
Manish V. Reddy, Andrzej Banburski, Nishka Pant +1
A convolutional neural network strongly robust to adversarial perturbations at reasonable computational and performance cost has not yet been demonstrated. The primate visual ventr…
Hierarchically Compositional Tasks and Deep Convolutional Networks
Arturo Deza, Qianli Liao, Andrzej Banburski +1
The main success stories of deep learning, starting with ImageNet, depend on deep convolutional networks, which on certain tasks perform significantly better than traditional shall…
Double descent in the condition number
Tomaso Poggio, Gil Kur, Andrzej Banburski
In solving a system of linear equations in variables , the condition number of the matrix measures how much errors in the data affect the solution .…
Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization
Tomaso Poggio, Andrzej Banburski, Qianli Liao
While deep learning is successful in a number of applications, it is not yet well understood theoretically. A satisfactory theoretical characterization of deep learning however, is…
Function approximation by deep networks
H. N. Mhaskar, T. Poggio
We show that deep networks are better than shallow networks at approximating functions that can be expressed as a composition of functions described by a directed acyclic graph, be…
Theory III: Dynamics and Generalization in Deep Networks
Andrzej Banburski, Qianli Liao, Brando Miranda +4
The key to generalization is controlling the complexity of the network. However, there is no obvious control of complexity -- such as an explicit regularization term -- in the trai…
Biologically-plausible learning algorithms can scale to large datasets
Will Xiao, Honglin Chen, Qianli Liao +1
The backpropagation (BP) algorithm is often thought to be biologically implausible in the brain. One of the main reasons is that BP requires symmetric weight matrices in the feedfo…
A Surprising Linear Relationship Predicts Test Performance in Deep Networks
Qianli Liao, Brando Miranda, Andrzej Banburski +2
Given two networks with the same training loss on a dataset, when would they have drastically different test losses and errors? Better understanding of this question of generalizat…
Theory IIIb: Generalization in Deep Networks
Tomaso Poggio, Qianli Liao, Brando Miranda +3
A main puzzle of deep neural networks (DNNs) revolves around the apparent absence of "overfitting", defined in this paper as follows: the expected error does not get worse when inc…
An analysis of training and generalization errors in shallow and deep networks
Hrushikesh Mhaskar, Tomaso Poggio
This paper is motivated by an open problem around deep networks, namely, the apparent absence of over-fitting despite large over-parametrization which allows perfect fitting of the…
Theory of Deep Learning III: explaining the non-overfitting puzzle
Tomaso Poggio, Kenji Kawaguchi, Qianli Liao +5
A main puzzle of deep networks revolves around the absence of overfitting despite large overparametrization and despite the large capacity demonstrated by zero training error on ra…
Theory of Deep Learning IIb: Optimization Properties of SGD
Chiyuan Zhang, Qianli Liao, Alexander Rakhlin +3
In Theory IIb we characterize with a mix of theory and experiments the optimization of deep convolutional networks by Stochastic Gradient Descent. The main new result in this paper…
Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin +1
We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity --- the Fisher-Rao nor…
Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review
Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco +2
The paper characterizes classes of functions for which deep learning can be exponentially better than shallow learning. Deep convolutional networks are a special case of these cond…
Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
Qianli Liao, Kenji Kawaguchi, Tomaso Poggio
We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major lim…
Deep vs. shallow networks : An approximation theory perspective
Hrushikesh Mhaskar, Tomaso Poggio
The paper briefy reviews several recent results on hierarchical architectures for learning from examples, that may formally explain the conditions under which Deep Convolutional Ne…
Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
Qianli Liao, Tomaso Poggio
We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a special type of shallo…
Learning Functions: When Is Deep Better Than Shallow
Hrushikesh Mhaskar, Qianli Liao, Tomaso Poggio
While the universal approximation property holds both for hierarchical and shallow networks, we prove that deep (hierarchical) networks can approximate the class of compositional f…
Foveation-based Mechanisms Alleviate Adversarial Examples
Yan Luo, Xavier Boix, Gemma Roig +2
We show that adversarial examples, i.e., the visually imperceptible perturbations that result in Convolutional Neural Networks (CNNs) fail, can be alleviated with a mechanism based…