179 citations · 294 across the 16 of their papers we have counts for
16 papers
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models
Aleksandar Botev, Soham De, Samuel L Smith +59
We introduce RecurrentGemma, a family of open language models which uses Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve ex…
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
Soham De, Samuel L. Smith, Anushan Fernando +14
Recurrent neural networks (RNNs) have fast inference and scale efficiently on long sequences, but they are difficult to train and hard to scale. We propose Hawk, an RNN with gated…
Applications of flow models to the generation of correlated lattice QCD ensembles
Ryan Abbott, Aleksandar Botev, Denis Boyda +7
Machine-learned normalizing flows can be used in the context of lattice quantum field theory to generate statistically correlated ensembles of lattice gauge fields at different act…
Normalizing flows for lattice gauge theory in arbitrary space-time dimension
Ryan Abbott, Michael S. Albergo, Aleksandar Botev +11
Applications of normalizing flows to the sampling of field configurations in lattice gauge theory have so far been explored almost exclusively in two space-time dimensions. We repo…
Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation
Bobby He, James Martens, Guodong Zhang +4
Skip connections and normalisation layers form two standard architectural components that are ubiquitous for the training of Deep Neural Networks (DNNs), but whose precise roles ar…
Aspects of scaling and scalability for flow-based sampling of lattice QCD
Ryan Abbott, Michael S. Albergo, Aleksandar Botev +10
Recent applications of machine-learned normalizing flows to sampling in lattice field theory suggest that such methods may be able to mitigate critical slowing down and topological…
Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers
Guodong Zhang, Aleksandar Botev, James Martens
Training very deep neural networks is still an extremely challenging task. The common solution is to use shortcut connections and normalization layers, which are both crucial ingre…
SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision
Irina Higgins, Peter Wirnsberger, Andrew Jaegle +1
A recently proposed class of models attempts to learn latent dynamics from high-dimensional observations, like images, using priors informed by Hamiltonian mechanics. While these m…
Which priors matter? Benchmarking models for learning latent dynamics
Aleksandar Botev, Andrew Jaegle, Peter Wirnsberger +2
Learning dynamics is at the heart of many important applications of machine learning (ML), such as robotics and autonomous driving. In these settings, ML algorithms typically need…
Better, Faster Fermionic Neural Networks
James S. Spencer, David Pfau, Aleksandar Botev +1
The Fermionic Neural Network (FermiNet) is a recently-developed neural network architecture that can be used as a wavefunction Ansatz for many-electron systems, and has already dem…
Disentangling by Subspace Diffusion
David Pfau, Irina Higgins, Aleksandar Botev +1
We present a novel nonparametric algorithm for symmetry-based disentangling of data manifolds, the Geometric Manifold Component Estimator (GEOMANCER). GEOMANCER provides a partial…
Hamiltonian Generative Networks
Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle +3
The Hamiltonian formalism plays a central role in classical and quantum physics. Hamiltonians are the main tool for modelling the continuous time evolution of systems with conserve…
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
Hippolyt Ritter, Aleksandar Botev, David Barber
We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning fra…
Practical Gauss-Newton Optimisation for Deep Learning
Aleksandar Botev, Hippolyt Ritter, David Barber
We present an efficient block-diagonal ap- proximation to the Gauss-Newton matrix for feedforward neural networks. Our result- ing algorithm is competitive against state- of-the-ar…
Nesterov's Accelerated Gradient and Momentum as approximations to Regularised Update Descent
Aleksandar Botev, Guy Lever, David Barber
We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nes…
Dealing with a large number of classes -- Likelihood, Discrimination or Ranking?
David Barber, Aleksandar Botev
We consider training probabilistic classifiers in the case of a large number of classes. The number of classes is assumed too large to perform exact normalisation over all classes.…