61 citations · 200 across the 9 of their papers we have counts for
6 papers · 1 filter
Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
James Martens, Andy Ballard, Guillaume Desjardins +4
Using an extended and formalized version of the Q/C map analysis of Poole et al. (2016), along with Neural Tangent Kernel theory, we identify the main pathologies present in deep n…
Sideways: Depth-Parallel Training of Video Models
Mateusz Malinowski, Grzegorz Swirszcz, Joao Carreira +1
We propose Sideways, an approximate backpropagation scheme for training video models. In standard backpropagation, the gradients and activations at every computation step through t…
Verification of Non-Linear Specifications for Neural Networks
Chongli Qin, Krishnamurthy, Dvijotham +7
Prior work on neural network verification has focused on specifications that are linear functions of the output of the network, e.g., invariance of the classifier output under adve…
Distilling Policy Distillation
Wojciech Marian Czarnecki, Razvan Pascanu, Simon Osindero +3
The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success,…
Sobolev Training for Neural Networks
Wojciech Marian Czarnecki, Simon Osindero, Max Jaderberg +2
At the heart of deep learning we aim to use neural networks as function approximators - training them to produce outputs from inputs in emulation of a ground truth function or data…
Understanding Synthetic Gradients and Decoupled Neural Interfaces
Wojciech Marian Czarnecki, Grzegorz Świrszcz, Max Jaderberg +3
When training neural networks, the use of Synthetic Gradients (SG) allows layers or modules to be trained without update locking - without waiting for a true error gradient to be b…