1 citations · 1 across the 7 of their papers we have counts for
Showing 2024Show all
2 papers · 1 filter
cs.LG2024
Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji +2
This paper questions whether the strong performance of softmax attention in transformers stems from producing a probability distribution over inputs. Instead, we argue that softmax…
cs.LG2024★ 1 cited
Efficient Learning With Sine-Activated Low-rank Matrices
Yiping Ji, Hemanth Saratchandran, Cameron Gordon +2
Low-rank decomposition has emerged as a vital tool for enhancing parameter efficiency in neural network architectures, gaining traction across diverse applications in machine learn…