14.1k citations · 34.6k across the 53 of their papers we have counts for
26 papers · 1 filter
The Forward-Forward Algorithm: Some Preliminary Investigations
Geoffrey Hinton
The aim of this paper is to introduce a new learning procedure for neural networks and to demonstrate that it works well enough on a few small problems to be worth further investig…
Gaussian-Bernoulli RBMs Without Tears
Renjie Liao, Simon Kornblith, Mengye Ren +2
We revisit the challenging problem of training Gaussian-Bernoulli restricted Boltzmann machines (GRBMs), introducing two innovations. We propose a novel Gibbs-Langevin sampling alg…
Scaling Forward Gradient With Local Losses
Mengye Ren, Simon Kornblith, Renjie Liao +1
Forward gradient learning computes a noisy directional gradient and is a biologically plausible alternative to backprop for learning deep neural networks. However, the standard for…
Teaching with Commentaries
Aniruddh Raghu, Maithra Raghu, Simon Kornblith +2
Effective training of deep neural networks can be challenging, and there remain many open questions on how to best learn these models. Recently developed methods to improve neural…
Big Self-Supervised Models are Strong Semi-Supervised Learners
Ting Chen, Simon Kornblith, Kevin Swersky +2
One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Althou…
Neural Additive Models: Interpretable Machine Learning with Neural Nets
Rishabh Agarwal, Levi Melnick, Nicholas Frosst +4
Deep neural networks (DNNs) are powerful black-box predictors that have achieved impressive performance on a wide variety of tasks. However, their accuracy comes at the cost of int…
Deflecting Adversarial Attacks
Yao Qin, Nicholas Frosst, Colin Raffel +2
There has been an ongoing cycle where stronger defenses against adversarial attacks are subsequently broken by a more advanced defense-aware attack. We present a new approach towar…
A Simple Framework for Contrastive Learning of Visual Representations
Ting Chen, Simon Kornblith, Mohammad Norouzi +1
This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms wit…
Subclass Distillation
Rafael Müller, Simon Kornblith, Geoffrey Hinton
After a large "teacher" neural network has been trained on labeled data, the probabilities that the teacher assigns to incorrect classes reveal a lot of information about the way i…
Lookahead Optimizer: k steps forward, 1 step back
Michael R. Zhang, James Lucas, Geoffrey Hinton +1
The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categ…
Detecting and Diagnosing Adversarial Images with Class-Conditional Capsule Reconstructions
Yao Qin, Nicholas Frosst, Sara Sabour +3
Adversarial examples raise questions about whether neural network models are sensitive to the same visual features as humans. In this paper, we first detect adversarial examples or…
When Does Label Smoothing Help?
Rafael Müller, Simon Kornblith, Geoffrey Hinton
The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and…
Learning Sparse Networks Using Targeted Dropout
Aidan N. Gomez, Ivan Zhang, Siddhartha Rao Kamalakara +4
Neural networks are easier to optimise when they have many more weights than are required for modelling the mapping from inputs to outputs. This suggests a two-stage learning proce…
Similarity of Neural Network Representations Revisited
Simon Kornblith, Mohammad Norouzi, Honglak Lee +1
Recent work has sought to understand the behavior of neural networks by comparing representations between layers and between different trained models. We examine methods for compar…
DARCCC: Detecting Adversaries by Reconstruction from Class Conditional Capsules
Nicholas Frosst, Sara Sabour, Geoffrey Hinton
We present a simple technique that allows capsule models to detect adversarial images. In addition to being trained to classify images, the capsule model is trained to reconstruct…
Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
Sergey Bartunov, Adam Santoro, Blake A. Richards +3
The backpropagation of error algorithm (BP) is impossible to implement in a real brain. The recent success of deep networks in machine learning and AI, however, has inspired propos…
Large scale distributed neural network training through online distillation
Rohan Anil, Gabriel Pereyra, Alexandre Passos +3
Techniques such as ensembling and distillation promise model quality improvements when paired with almost any base model. However, due to increased test-time cost (for ensembles) a…
Distilling a Neural Network Into a Soft Decision Tree
Nicholas Frosst, Geoffrey Hinton
Deep neural networks have proved to be a very effective way to perform classification tasks. They excel when the input data is high dimensional, the relationship between the input…
Who Said What: Modeling Individual Labelers Improves Classification
Melody Y. Guan, Varun Gulshan, Andrew M. Dai +1
Data are often labeled by many different experts with each expert only labeling a small fraction of the data and each data point being labeled by several experts. This reduces the…
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz +4
The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis…
Modeling Documents with Deep Boltzmann Machines
Nitish Srivastava, Ruslan R Salakhutdinov, Geoffrey E. Hinton
We introduce a Deep Boltzmann Machine model suitable for modeling and extracting latent semantic representations from a large unstructured collection of documents. We overcome the…
Discovering Multiple Constraints that are Frequently Approximately Satisfied
Geoffrey E. Hinton, Yee Whye Teh
Some high-dimensional data.sets can be modelled by assuming that there are many different linear constraints, each of which is Frequently Approximately Satisfied (FAS) by the data.…
Efficient Parametric Projection Pursuit Density Estimation
Max Welling, Richard S. Zemel, Geoffrey E. Hinton
Product models of low dimensional experts are a powerful way to avoid the curse of dimensionality. We present the ``under-complete product of experts' (UPoE), where each expert mod…
Deep Mixtures of Factor Analysers
Yichuan Tang, Ruslan Salakhutdinov, Geoffrey Hinton
An efficient way to learn deep density models that have many layers of latent variables is to learn one layer at a time using a model that has only one layer of latent variables. A…