258 citations · 1.1k across the 17 of their papers we have counts for
18 papers
Exploring Low Rank Training of Deep Neural Networks
Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh +3
Training deep neural networks in low rank, i.e. with factorised layers, is of particular interest to the community: it offers efficiency over unfactorised training in terms of both…
Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt
Sören Mindermann, Jan Brauner, Muhammed Razzak +8
Training on web-scale data can take months. But most computation and time is wasted on redundant and noisy points that are already learnt or not learnable. To accelerate training,…
Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval
Pascal Notin, Mafalda Dias, Jonathan Frazer +4
The ability to accurately model the fitness landscape of protein sequences is critical to a wide range of applications, from quantifying the effects of human variants on disease li…
Prioritized training on points that are learnable, worth learning, and not yet learned (workshop version)
Sören Mindermann, Muhammed Razzak, Winnie Xu +7
We introduce Goldilocks Selection, a technique for faster model training which selects a sequence of training points that are "just right". We propose an information-theoretic acqu…
Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning
Jannik Kossen, Neil Band, Clare Lyle +3
We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To t…
Robustness to Pruning Predicts Generalization in Deep Neural Networks
Lorenz Kuhn, Clare Lyle, Aidan N. Gomez +2
Existing generalization measures that aim to capture a model's simplicity based on parameter counts or norms fail to explain generalization in overparameterized deep neural network…
Interlocking Backpropagation: Improving depthwise model-parallelism
Aidan N. Gomez, Oscar Key, Kuba Perlin +4
The number of parameters in state of the art neural networks has drastically increased in recent years. This surge of interest in large scale neural networks has motivated the deve…
Improving compute efficacy frontiers with SliceOut
Pascal Notin, Aidan N. Gomez, Joanna Yoo +1
Pushing forward the compute efficacy frontier in deep learning is critical for tasks that require frequent model re-training or workloads that entail training a large number of mod…
Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers
Tim Z. Xiao, Aidan N. Gomez, Yarin Gal
We detect out-of-training-distribution sentences in Neural Machine Translation using the Bayesian Deep Learning equivalent of Transformer models. For this we develop a new measure…
A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
Angelos Filos, Sebastian Farquhar, Aidan N. Gomez +6
Evaluation of Bayesian deep learning (BDL) methods is challenging. We often seek to evaluate the methods' robustness and scalability, assessing whether new tools give `better' unce…
The Difficulty of Training Sparse Neural Networks
Utku Evci, Fabian Pedregosa, Aidan Gomez +1
We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime. Recent w…
Learning Sparse Networks Using Targeted Dropout
Aidan N. Gomez, Ivan Zhang, Siddhartha Rao Kamalakara +4
Neural networks are easier to optimise when they have many more weights than are required for modelling the mapping from inputs to outputs. This suggests a two-stage learning proce…
Tensor2Tensor for Neural Machine Translation
Ashish Vaswani, Samy Bengio, Eugene Brevdo +10
Tensor2Tensor is a library for deep learning models that is well-suited for neural machine translation and includes the reference implementation of the state-of-the-art Transformer…
Unsupervised Cipher Cracking Using Discrete GANs
Aidan N. Gomez, Sicong Huang, Ivan Zhang +3
This work details CipherGAN, an architecture inspired by CycleGAN used for inferring the underlying cipher mapping given banks of unpaired ciphertext and plaintext. We demonstrate…
The Reversible Residual Network: Backpropagation Without Storing Activations
Aidan N. Gomez, Mengye Ren, Raquel Urtasun +1
Deep residual networks (ResNets) have significantly pushed forward the state-of-the-art on image classification, increasing in performance as networks grow both deeper and wider. H…
One Model To Learn Them All
Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer +4
Deep learning yields great results across many fields, from speech recognition, image classification, to translation. But for each problem, getting a deep model to work well involv…
Depthwise Separable Convolutions for Neural Machine Translation
Lukasz Kaiser, Aidan N. Gomez, Francois Chollet
Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency. They have been shown…
Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar +5
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also conne…