4 papers
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
Harry Jake Cunningham, Nicola Muca Cirone
Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models (LLMs). However, this approach has significant limitations…
Infinite Neural Operators: Gaussian processes on functions
Daniel Augusto de Souza, Yuchen Zhu, Harry Jake Cunningham +3
A variety of infinitely wide neural architectures (e.g., dense NNs, CNNs, and transformers) induce Gaussian process (GP) priors over their outputs. These relationships provide both…
Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling
Harry Jake Cunningham, Giorgio Giannone, Mingtian Zhang +1
Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must…
RDM: Recurrent Diffusion Model for Human Motion Generation
Mirgahney Mohamed, Harry Jake Cunningham, Marc P. Deisenroth +1
Human motion generation is a challenging task due to its high dimensionality and the difficulty of generating fine-grained motions. Diffusion methods have been proposed due to thei…