101 citations · 389 across the 29 of their papers we have counts for
3 papers · 1 filter
Zero-th Order Algorithm for Softmax Attention Optimization
Yichuan Deng, Zhihang Li, Sridhar Mahadevan +1
Large language models (LLMs) have brought about significant transformations in human society. Among the crucial computations in LLMs, the softmax unit holds great importance. Its h…
Randomized and Deterministic Attention Sparsification Algorithms for Over-parameterized Feature Dimension
Yichuan Deng, Sridhar Mahadevan, Zhao Song
Large language models (LLMs) have shown their power in different areas. Attention computation, as an important subroutine of LLMs, has also attracted interests in theory. Recently…
An Over-parameterized Exponential Regression
Yeqi Gao, Sridhar Mahadevan, Zhao Song
Over the past few years, there has been a significant amount of research focused on studying the ReLU activation function, with the aim of achieving neural network convergence thro…