26 citations · 86 across the 21 of their papers we have counts for
13 papers · 1 filter
Hessian Spectral Analysis at Foundation Model Scale
Diego Granziol, Khurshid Juarev
Accurate Hessian spectra of foundation models have remained out of reach, leading most prior work to rely on small models or strong structural approximations. We show that faithful…
A Linear Approach to Data Poisoning
Donald Flynn, Diego Granziol
Backdoor and data-poisoning attacks can flip predictions with tiny training corruptions, yet a sharp theory linking poisoning strength, overparameterization, and regularization is…
A Random Matrix Theory Approach to Damping in Deep Learning
Diego Granziol, Nicholas Baskerville
We conjecture that the inherent difference in generalisation between adaptive and non-adaptive gradient methods in deep learning stems from the increased estimation noise in the fl…
Flatness is a False Friend
Diego Granziol
Hessian based measures of flatness, such as the trace, Frobenius and spectral norms, have been argued, used and shown to relate to generalisation. In this paper we demonstrate that…
Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
Diego Granziol, Stefan Zohren, Stephen Roberts
We study the effect of mini-batching on the loss landscape of deep neural networks using spiked, field-dependent random matrix theory. We demonstrate that the magnitude of the extr…
Beyond Random Matrix Theory for Deep Networks
Diego Granziol
We investigate whether the Wigner semi-circle and Marcenko-Pastur distributions, often used for deep neural network theoretical analysis, match empirically observed spectral densit…
Iterative Averaging in the Quest for Best Test Error
Diego Granziol, Xingchen Wan, Samuel Albanie +1
We analyse and explain the increased generalisation performance of iterate averaging using a Gaussian process perturbation model between the true and batch risk surface on the high…
A Maximum Entropy approach to Massive Graph Spectra
Diego Granziol, Robin Ru, Stefan Zohren +3
Graph spectral techniques for measuring graph similarity, or for learning the cluster number, require kernel smoothing. The choice of kernel function and bandwidth are typically ch…
Deep Curvature Suite
Diego Granziol, Xingchen Wan, Timur Garipov
We present MLRG Deep Curvature suite, a PyTorch-based, open-source package for analysis and visualisation of neural network curvature and loss landscape. Despite of providing rich…
MEMe: An Accurate Maximum Entropy Method for Efficient Approximations in Large-Scale Machine Learning
Diego Granziol, Binxin Ru, Stefan Zohren +3
Efficient approximation lies at the heart of large-scale machine learning problems. In this paper, we propose a novel, robust maximum entropy algorithm, which is capable of dealing…
Entropic Spectral Learning for Large-Scale Graphs
Diego Granziol, Binxin Ru, Stefan Zohren +3
Graph spectra have been successfully used to classify network types, compute the similarity between graphs, and determine the number of communities in a network. For large graphs,…
Fast Information-theoretic Bayesian Optimisation
Binxin Ru, Mark McLeod, Diego Granziol +1
Information-theoretic Bayesian optimisation techniques have demonstrated state-of-the-art performance in tackling important global optimisation problems. However, current informati…
Entropic Determinants
Diego Granziol, Stephen Roberts
The ability of many powerful machine learning algorithms to deal with large data sets without compromise is often hampered by computationally expensive linear algebra tasks, of whi…