442 citations · 1.6k across the 20 of their papers we have counts for
21 papers
Should We Skip Diffusion?
Yiping Ji, James Martens, Simon Lucey
Diffusion models learn semantic representations while generating images. In the Decoupled Diffusion Transformer (DDT), a condition encoder provides features that guide a velocity d…
Orthogonal Self-Attention
Leo Zhang, James Martens
Softmax Self-Attention (SSA) is a key component of Transformer architectures. However, when utilised within skipless architectures, which aim to improve representation learning, re…
Cutting the Skip: Training Residual-Free Transformers
Yiping Ji, James Martens, Jianqiao Zheng +5
Transformers have achieved remarkable success across a wide range of applications, a feat often attributed to their scalability. Yet training them without skip (residual) connectio…
Discovery of Unstable Singularities
Yongji Wang, Mehdi Bennani, James Martens +19
Whether singularities can form in fluids remains a foundational unanswered question in mathematics. This phenomenon occurs when solutions to governing equations, such as the 3D Eul…
Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation
Bobby He, James Martens, Guodong Zhang +4
Skip connections and normalisation layers form two standard architectural components that are ubiquitous for the training of Deep Neural Networks (DNNs), but whose precise roles ar…
Pre-training via Denoising for Molecular Property Prediction
Sheheryar Zaidi, Michael Schaarschmidt, James Martens +6
Many important problems involving molecular property prediction from 3D structures have limited data, posing a generalization challenge for neural networks. In this paper, we descr…
Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers
Guodong Zhang, Aleksandar Botev, James Martens
Training very deep neural networks is still an extremely challenging task. The common solution is to use shortcut connections and normalization layers, which are both crucial ingre…
Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
James Martens, Andy Ballard, Guillaume Desjardins +4
Using an extended and formalized version of the Q/C map analysis of Poole et al. (2016), along with Neural Tangent Kernel theory, we identify the main pathologies present in deep n…
On the validity of kernel approximations for orthogonally-initialized neural networks
James Martens
In this note we extend kernel function approximation results for neural networks with Gaussian-distributed weights to single-layer networks initialized using Haar-distributed rando…
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
Guodong Zhang, Lala Li, Zachary Nado +5
Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns. In this work, we…
Adversarial Robustness through Local Linearization
Chongli Qin, James Martens, Sven Gowal +6
Adversarial training is an effective methodology for training deep neural networks that are robust against adversarial, norm-bounded perturbations. However, the computational cost…
Differentiable Game Mechanics
Alistair Letcher, David Balduzzi, Sebastien Racaniere +4
Deep learning is built on the foundational guarantee that gradient descent on an objective function converges to local minima. Unfortunately, this guarantee fails in settings, such…
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
Guodong Zhang, James Martens, Roger Grosse
Natural gradient descent has proven effective at mitigating the effects of pathological curvature in neural network optimization, but little is known theoretically about its conver…
On the Variance of Unbiased Online Recurrent Optimization
Tim Cooijmans, James Martens
The recently proposed Unbiased Online Recurrent Optimization algorithm (UORO, arXiv:1702.05043) uses an unbiased approximation of RTRL to achieve fully online gradient-based learni…
The Mechanics of n-Player Differentiable Games
David Balduzzi, Sebastien Racaniere, James Martens +3
The cornerstone underpinning deep learning is the guarantee that gradient descent on an objective converges to local minima. Unfortunately, this guarantee fails in settings, such a…
A Kronecker-factored approximate Fisher matrix for convolution layers
Roger Grosse, James Martens
Second-order optimization methods such as natural gradient descent have the potential to speed up training of neural networks by correcting for the curvature of the loss function.…
Adding Gradient Noise Improves Learning for Very Deep Networks
Arvind Neelakantan, Luke Vilnis, Quoc V. Le +4
Deep feedforward and recurrent networks have achieved impressive results in many perception and language processing applications. This success is partially attributed to architectu…
Optimizing Neural Networks with Kronecker-factored Approximate Curvature
James Martens, Roger Grosse
We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-Factored Approximate Curvature (K-FAC). K-FAC is based on an ef…
New insights and perspectives on the natural gradient method
James Martens
Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to s…
On the Expressive Efficiency of Sum Product Networks
James Martens, Venkatesh Medabalimi
Sum Product Networks (SPNs) are a recently developed class of deep generative models which compute their associated unnormalized density functions using a special type of arithmeti…
Estimating the Hessian by Back-propagating Curvature
James Martens, Ilya Sutskever, Kevin Swersky
In this work we develop Curvature Propagation (CP), a general technique for efficiently computing unbiased approximations of the Hessian of any function that is computed using a co…