17 citations · 32 across the 11 of their papers we have counts for
16 papers · 1 filter
Dynamic Stashing Quantization for Efficient Transformer Training
Guo Yang, Daniel Lo, Robert Mullins +1
Large Language Models (LLMs) have demonstrated impressive performance on a range of Natural Language Processing (NLP) tasks. Unfortunately, the immense amount of computations and m…
Wide Attention Is The Way Forward For Transformers?
Jason Ross Brown, Yiren Zhao, Ilia Shumailov +1
The Transformer is an extremely powerful and prominent deep learning architecture. In this work, we challenge the commonly held belief in deep learning that going deeper is better,…
Revisiting Structured Dropout
Yiren Zhao, Oluwatomisin Dada, Xitong Gao +1
Large neural networks are often overparameterised and prone to overfitting, Dropout is a widely used regularization technique to combat overfitting and improve model generalization…
DARTFormer: Finding The Best Type Of Attention
Jason Ross Brown, Yiren Zhao, Ilia Shumailov +1
Given the wide and ever growing range of different efficient Transformer attention mechanisms, it is important to identify which attention is most effective when given a task. In t…
Augmentation Backdoors
Joseph Rance, Yiren Zhao, Ilia Shumailov +1
Data augmentation is used extensively to improve model generalisation. However, reliance on external libraries to implement augmentation methods introduces a vulnerability into the…
Model Architecture Adaption for Bayesian Neural Networks
Duo Wang, Yiren Zhao, Ilia Shumailov +1
Bayesian Neural Networks (BNNs) offer a mathematically grounded framework to quantify the uncertainty of model predictions but come with a prohibitive computation cost for both tra…