353 citations · 1.4k across the 27 of their papers we have counts for
5 papers · 1 filter
Three Factors Influencing Minima in SGD
Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit +4
We investigate the dynamical and convergent properties of stochastic gradient descent (SGD) applied to Deep Neural Networks (DNNs). Characterizing the relation between learning rat…
Residual Connections Encourage Iterative Inference
Stanisław Jastrzębski, Devansh Arpit, Nicolas Ballas +3
Residual networks (Resnets) have become a prominent architecture in deep learning. However, a comprehensive understanding of Resnets is still a topic of ongoing research. A recent…
A Closer Look at Memorization in Deep Networks
Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas +8
We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noi…
Learning to Compute Word Embeddings On the Fly
Dzmitry Bahdanau, Tom Bosc, Stanisław Jastrzębski +3
Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. Learning representations for words in the "long tail" of this distributio…
How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks
Stanisław Jastrzebski, Damian Leśniak, Wojciech Marian Czarnecki
Maybe the single most important goal of representation learning is making subsequent learning faster. Surprisingly, this fact is not well reflected in the way embeddings are evalua…