297 citations · 358 across the 8 of their papers we have counts for
1 paper · 1 filter
Mitchell Wortsman, Jaehoon Lee, Justin Gilmer +1
Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU. In the context of vision transformers, we find that…