1 citations · 1 across the 18 of their papers we have counts for
Showing 2024Show all
2 papers · 1 filter
cs.LG2024
Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji +2
This paper questions whether the strong performance of softmax attention in transformers stems from producing a probability distribution over inputs. Instead, we argue that softmax…
cs.CV2024★ 1 cited
Structured Initialization for Attention in Vision Transformers
Jianqiao Zheng, Xueqian Li, Simon Lucey
The training of vision transformer (ViT) networks on small-scale datasets poses a significant challenge. By contrast, convolutional neural networks (CNNs) have an architectural ind…