17 citations · 32 across the 11 of their papers we have counts for
21 papers
Wide Attention Is The Way Forward For Transformers?
Jason Ross Brown, Yiren Zhao, Ilia Shumailov +1
The Transformer is an extremely powerful and prominent deep learning architecture. In this work, we challenge the commonly held belief in deep learning that going deeper is better,…
Revisiting Structured Dropout
Yiren Zhao, Oluwatomisin Dada, Xitong Gao +1
Large neural networks are often overparameterised and prone to overfitting, Dropout is a widely used regularization technique to combat overfitting and improve model generalization…
DARTFormer: Finding The Best Type Of Attention
Jason Ross Brown, Yiren Zhao, Ilia Shumailov +1
Given the wide and ever growing range of different efficient Transformer attention mechanisms, it is important to identify which attention is most effective when given a task. In t…
Augmentation Backdoors
Joseph Rance, Yiren Zhao, Ilia Shumailov +1
Data augmentation is used extensively to improve model generalisation. However, reliance on external libraries to implement augmentation methods introduces a vulnerability into the…
Model Architecture Adaption for Bayesian Neural Networks
Duo Wang, Yiren Zhao, Ilia Shumailov +1
Bayesian Neural Networks (BNNs) offer a mathematically grounded framework to quantify the uncertainty of model predictions but come with a prohibitive computation cost for both tra…
DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning
Robert Hönig, Yiren Zhao, Robert Mullins
Federated Learning (FL) is a powerful technique for training a model on a server with data from several clients in a privacy-preserving manner. In FL, a server sends the model to e…