185 citations · 657 across the 21 of their papers we have counts for
15 papers · 1 filter
Explicit regularization and implicit bias in deep network classifiers trained with the square loss
Tomaso Poggio, Qianli Liao
Deep ReLU networks trained with the square loss have been observed to perform well in classification tasks. We provide here a theoretical justification based on analysis of the ass…
Hierarchically Compositional Tasks and Deep Convolutional Networks
Arturo Deza, Qianli Liao, Andrzej Banburski +1
The main success stories of deep learning, starting with ImageNet, depend on deep convolutional networks, which on certain tasks perform significantly better than traditional shall…
Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization
Tomaso Poggio, Andrzej Banburski, Qianli Liao
While deep learning is successful in a number of applications, it is not yet well understood theoretically. A satisfactory theoretical characterization of deep learning however, is…
Theory III: Dynamics and Generalization in Deep Networks
Andrzej Banburski, Qianli Liao, Brando Miranda +4
The key to generalization is controlling the complexity of the network. However, there is no obvious control of complexity -- such as an explicit regularization term -- in the trai…
Biologically-plausible learning algorithms can scale to large datasets
Will Xiao, Honglin Chen, Qianli Liao +1
The backpropagation (BP) algorithm is often thought to be biologically implausible in the brain. One of the main reasons is that BP requires symmetric weight matrices in the feedfo…
A Surprising Linear Relationship Predicts Test Performance in Deep Networks
Qianli Liao, Brando Miranda, Andrzej Banburski +2
Given two networks with the same training loss on a dataset, when would they have drastically different test losses and errors? Better understanding of this question of generalizat…
Theory IIIb: Generalization in Deep Networks
Tomaso Poggio, Qianli Liao, Brando Miranda +3
A main puzzle of deep neural networks (DNNs) revolves around the apparent absence of "overfitting", defined in this paper as follows: the expected error does not get worse when inc…
Theory of Deep Learning III: explaining the non-overfitting puzzle
Tomaso Poggio, Kenji Kawaguchi, Qianli Liao +5
A main puzzle of deep networks revolves around the absence of overfitting despite large overparametrization and despite the large capacity demonstrated by zero training error on ra…
Theory of Deep Learning IIb: Optimization Properties of SGD
Chiyuan Zhang, Qianli Liao, Alexander Rakhlin +3
In Theory IIb we characterize with a mix of theory and experiments the optimization of deep convolutional networks by Stochastic Gradient Descent. The main new result in this paper…
Theory II: Landscape of the Empirical Risk in Deep Learning
Qianli Liao, Tomaso Poggio
Previous theoretical work on deep learning and neural network optimization tend to focus on avoiding saddle points and local minima. However, the practical observation is that, at…
Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review
Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco +2
The paper characterizes classes of functions for which deep learning can be exponentially better than shallow learning. Deep convolutional networks are a special case of these cond…
Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
Qianli Liao, Kenji Kawaguchi, Tomaso Poggio
We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major lim…
Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
Qianli Liao, Tomaso Poggio
We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a special type of shallo…
Learning Functions: When Is Deep Better Than Shallow
Hrushikesh Mhaskar, Qianli Liao, Tomaso Poggio
While the universal approximation property holds both for hierarchical and shallow networks, we prove that deep (hierarchical) networks can approximate the class of compositional f…
How Important is Weight Symmetry in Backpropagation?
Qianli Liao, Joel Z. Leibo, Tomaso Poggio
Gradient backpropagation (BP) requires symmetric feedforward and feedback connections -- the same weights must be used for forward and backward passes. This "weight transport probl…