26 citations · 86 across the 21 of their papers we have counts for
21 papers
Hessian Spectral Analysis at Foundation Model Scale
Diego Granziol, Khurshid Juarev
Accurate Hessian spectra of foundation models have remained out of reach, leading most prior work to rely on small models or strong structural approximations. We show that faithful…
HessFormer: Hessians at Foundation Scale
Diego Granziol
Whilst there have been major advancements in the field of first order optimisation of deep learning models, where state of the art open source mixture of expert models go into the…
A Linear Approach to Data Poisoning
Donald Flynn, Diego Granziol
Backdoor and data-poisoning attacks can flip predictions with tiny training corruptions, yet a sharp theory linking poisoning strength, overparameterization, and regularization is…
Compute-Optimal LLMs Provably Generalize Better With Scale
Marc Finzi, Sanyam Kapoor, Diego Granziol +4
Why do larger language models generalize better? To investigate this question, we develop generalization bounds on the pretraining objective of large language models (LLMs) in the…
Universal characteristics of deep neural network loss surfaces from random matrix theory
Nicholas P Baskerville, Jonathan P Keating, Francesco Mezzadri +2
This paper considers several aspects of random matrix universality in deep neural networks. Motivated by recent experimental work, we use universal properties of random matrices re…
Ranker-agnostic Contextual Position Bias Estimation
Oriol Barbany Mayor, Vito Bellini, Alexander Buchholz +4
Learning-to-rank (LTR) algorithms are ubiquitous and necessary to explore the extensive catalogs of media providers. To avoid the user examining all the results, its preferences ar…
Appearance of Random Matrix Theory in Deep Learning
Nicholas P Baskerville, Diego Granziol, Jonathan P Keating
We investigate the local spectral statistics of the loss surface Hessians of artificial neural networks, where we discover excellent agreement with Gaussian Orthogonal Ensemble sta…
A Random Matrix Theory Approach to Damping in Deep Learning
Diego Granziol, Nicholas Baskerville
We conjecture that the inherent difference in generalisation between adaptive and non-adaptive gradient methods in deep learning stems from the increased estimation noise in the fl…
Flatness is a False Friend
Diego Granziol
Hessian based measures of flatness, such as the trace, Frobenius and spectral norms, have been argued, used and shown to relate to generalisation. In this paper we demonstrate that…
Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
Diego Granziol, Stefan Zohren, Stephen Roberts
We study the effect of mini-batching on the loss landscape of deep neural networks using spiked, field-dependent random matrix theory. We demonstrate that the magnitude of the extr…
Beyond Random Matrix Theory for Deep Networks
Diego Granziol
We investigate whether the Wigner semi-circle and Marcenko-Pastur distributions, often used for deep neural network theoretical analysis, match empirically observed spectral densit…
Iterative Averaging in the Quest for Best Test Error
Diego Granziol, Xingchen Wan, Samuel Albanie +1
We analyse and explain the increased generalisation performance of iterate averaging using a Gaussian process perturbation model between the true and batch risk surface on the high…
A Maximum Entropy approach to Massive Graph Spectra
Diego Granziol, Robin Ru, Stefan Zohren +3
Graph spectral techniques for measuring graph similarity, or for learning the cluster number, require kernel smoothing. The choice of kernel function and bandwidth are typically ch…
Deep Curvature Suite
Diego Granziol, Xingchen Wan, Timur Garipov
We present MLRG Deep Curvature suite, a PyTorch-based, open-source package for analysis and visualisation of neural network curvature and loss landscape. Despite of providing rich…
MEMe: An Accurate Maximum Entropy Method for Efficient Approximations in Large-Scale Machine Learning
Diego Granziol, Binxin Ru, Stefan Zohren +3
Efficient approximation lies at the heart of large-scale machine learning problems. In this paper, we propose a novel, robust maximum entropy algorithm, which is capable of dealing…
Entropic Spectral Learning for Large-Scale Graphs
Diego Granziol, Binxin Ru, Stefan Zohren +3
Graph spectra have been successfully used to classify network types, compute the similarity between graphs, and determine the number of communities in a network. For large graphs,…
VBALD - Variational Bayesian Approximation of Log Determinants
Diego Granziol, Edward Wagstaff, Bin Xin Ru +2
Evaluating the log determinant of a positive definite matrix is ubiquitous in machine learning. Applications thereof range from Gaussian processes, minimum-volume ellipsoids, metri…
Fast Information-theoretic Bayesian Optimisation
Binxin Ru, Mark McLeod, Diego Granziol +1
Information-theoretic Bayesian optimisation techniques have demonstrated state-of-the-art performance in tackling important global optimisation problems. However, current informati…
Entropic Determinants
Diego Granziol, Stephen Roberts
The ability of many powerful machine learning algorithms to deal with large data sets without compromise is often hampered by computationally expensive linear algebra tasks, of whi…
Entropic Trace Estimates for Log Determinants
Jack Fitzsimons, Diego Granziol, Kurt Cutajar +3
The scalable calculation of matrix determinants has been a bottleneck to the widespread application of many machine learning methods such as determinantal point processes, Gaussian…
An information and field theoretic approach to the grand canonical ensemble
Diego Granziol, Stephen Roberts
We present a novel derivation of the constraints required to obtain the underlying principles of statistical mechanics using a maximum entropy framework. We derive the mean value c…