430 citations · 471 across the 10 of their papers we have counts for
11 papers · 1 filter
Gaussian-Bernoulli RBMs Without Tears
Renjie Liao, Simon Kornblith, Mengye Ren +2
We revisit the challenging problem of training Gaussian-Bernoulli restricted Boltzmann machines (GRBMs), introducing two innovations. We propose a novel Gibbs-Langevin sampling alg…
On the Origins of the Block Structure Phenomenon in Neural Network Representations
Thao Nguyen, Maithra Raghu, Simon Kornblith
Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations. This block s…
Meta-Learning to Improve Pre-Training
Aniruddh Raghu, Jonathan Lorraine, Simon Kornblith +2
Pre-training (PT) followed by fine-tuning (FT) is an effective method for training neural networks, and has led to significant performance improvements in many domains. PT can inco…
Teaching with Commentaries
Aniruddh Raghu, Maithra Raghu, Simon Kornblith +2
Effective training of deep neural networks can be challenging, and there remain many open questions on how to best learn these models. Recently developed methods to improve neural…
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
Thao Nguyen, Maithra Raghu, Simon Kornblith
A key factor in the success of deep neural networks is the ability to scale models to improve performance by varying the architecture depth and width. This simple property of neura…
Big Self-Supervised Models are Strong Semi-Supervised Learners
Ting Chen, Simon Kornblith, Kevin Swersky +2
One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Althou…