Publications (21)
Convolutional Bipartite Attractor Networks
Michael Iuzzolino, Yoram Singer, Michael C. Mozer
In human perception and cognition, a fundamental operation that brains perform is interpretation: constructing coherent neural states from noisy, incomplete, and intrinsically ambi…
Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
Amit Daniely, Roy Frostig, Yoram Singer
We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations gener…
The Maximum Entropy Relaxation Path
Moshe Dubiner, Matan Gavish, Yoram Singer
The relaxed maximum entropy problem is concerned with finding a probability distribution on a finite set that minimizes the relative entropy to a given prior distribution, while sa…
Update Rules for Parameter Estimation in Bayesian Networks
Eric Bauer, Daphne Koller, Yoram Singer
This paper re-examines the problem of parameter estimation in Bayesian networks with missing values and hidden variables from the perspective of recent work in on-line learning [Ki…
Beyond Word N-Grams
Fernando C. N. Pereira, Yoram Singer, Naftali Tishby
We describe, analyze, and evaluate experimentally a new probabilistic model for word-sequence prediction in natural language based on prediction suffix trees (PSTs). By using effic…
Sketching and Neural Networks
Amit Daniely, Nevena Lazic, Yoram Singer +1
High-dimensional sparse data present computational and statistical challenges for supervised learning. We propose compact linear sketches for reducing the dimensionality of the inp…
Using Web Co-occurrence Statistics for Improving Image Categorization
Samy Bengio, Jeff Dean, Dumitru Erhan +5
Object recognition and localization are important tasks in computer vision. The focus of this work is the incorporation of contextual information in order to improve object recogni…
Proximity Preserving Binary Code using Signed Graph-Cut
Inbal Lav, Shai Avidan, Yoram Singer +1
We introduce a binary embedding framework, called Proximity Preserving Code (PPC), which learns similarity and dissimilarity between data points to create a compact and affinity-pr…
Are All Layers Created Equal?
Chiyuan Zhang, Samy Bengio, Yoram Singer
Understanding deep neural networks is a major research objective with notable experimental and theoretical attention in recent years. The practical success of excessively large net…
Train faster, generalize better: Stability of stochastic gradient descent
Moritz Hardt, Benjamin Recht, Yoram Singer
We show that parametric models trained by a stochastic gradient method (SGM) with few iterations have vanishing generalization error. We prove our results by arguing that SGM is al…
A Unified Approach to Adaptive Regularization in Online and Stochastic Optimization
Vineet Gupta, Tomer Koren, Yoram Singer
We describe a framework for deriving and analyzing online optimization algorithms that incorporate adaptive, data-dependent regularization, also termed preconditioning. Such algori…
Matrix Approximation under Local Low-Rank Assumption
Joonseok Lee, Seungyeon Kim, Guy Lebanon +1
Matrix approximation is a common tool in machine learning for building accurate prediction models for recommendation systems, text mining, and computer vision. A prevalent assumpti…
Random Features for Compositional Kernels
Amit Daniely, Roy Frostig, Vineet Gupta +1
We describe and analyze a simple random feature scheme (RFS) from prescribed compositional kernels. The compositional kernels we use are inspired by the structure of convolutional…
The Well Tempered Lasso
Yuanzhi Li, Yoram Singer
We study the complexity of the entire regularization path for least squares regression with 1-norm penalty, known as the Lasso. Every regression parameter in the Lasso changes line…
Switching Portfolios
Yoram Singer
A constant rebalanced portfolio is an asset allocation algorithm which keeps the same distribution of wealth among a set of assets along a period of time. Recently, there has been…
Shampoo: Preconditioned Stochastic Tensor Optimization
Vineet Gupta, Tomer Koren, Yoram Singer
Preconditioned gradient methods are among the most general and powerful tools in optimization. However, preconditioning requires storing and manipulating prohibitively large matric…
Memory-Efficient Adaptive Optimization
Rohan Anil, Vineet Gupta, Tomer Koren +1
Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling. However, these meth…
Identity Crisis: Memorization and Generalization under Extreme Overparameterization
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
We study the interplay between memorization and generalization of overparameterized networks in the extreme case of a single training example and an identity-mapping task. We exami…
Zero-Shot Learning by Convex Combination of Semantic Embeddings
Mohammad Norouzi, Tomas Mikolov, Samy Bengio +5
Several recent publications have proposed methods for mapping images into continuous semantic embedding spaces. In some cases the embedding space is trained jointly with the image…
Scalable Second Order Optimization for Deep Learning
Rohan Anil, Vineet Gupta, Tomer Koren +2
Optimization in machine learning, both theoretical and applied, is presently dominated by first-order gradient methods such as stochastic gradient descent. Second-order optimizatio…
Exponentiated Gradient Meets Gradient Descent
Udaya Ghai, Elad Hazan, Yoram Singer
The (stochastic) gradient descent and the multiplicative update method are probably the most popular algorithms in machine learning. We introduce and study a new regularization whi…