30 citations · 31 across the 10 of their papers we have counts for
3 papers · 1 filter
Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji +2
This paper questions whether the strong performance of softmax attention in transformers stems from producing a probability distribution over inputs. Instead, we argue that softmax…
Structured Initialization for Attention in Vision Transformers
Jianqiao Zheng, Xueqian Li, Simon Lucey
The training of vision transformer (ViT) networks on small-scale datasets poses a significant challenge. By contrast, convolutional neural networks (CNNs) have an architectural ind…
Convolutional Initialization for Data-Efficient Vision Transformers
Jianqiao Zheng, Xueqian Li, Simon Lucey
Training vision transformer networks on small datasets poses challenges. In contrast, convolutional neural networks (CNNs) can achieve state-of-the-art performance by leveraging th…