101 citations · 389 across the 29 of their papers we have counts for
29 papers
Zero-th Order Algorithm for Softmax Attention Optimization
Yichuan Deng, Zhihang Li, Sridhar Mahadevan +1
Large language models (LLMs) have brought about significant transformations in human society. Among the crucial computations in LLMs, the softmax unit holds great importance. Its h…
Randomized and Deterministic Attention Sparsification Algorithms for Over-parameterized Feature Dimension
Yichuan Deng, Sridhar Mahadevan, Zhao Song
Large language models (LLMs) have shown their power in different areas. Attention computation, as an important subroutine of LLMs, has also attracted interests in theory. Recently…
An Over-parameterized Exponential Regression
Yeqi Gao, Sridhar Mahadevan, Zhao Song
Over the past few years, there has been a significant amount of research focused on studying the ReLU activation function, with the aim of achieving neural network convergence thro…
A Layered Architecture for Universal Causality
Sridhar Mahadevan
We propose a layered hierarchical architecture called UCLA (Universal Causality Layered Architecture), which combines multiple levels of categorical abstraction for causal inferenc…
Unifying Causal Inference and Reinforcement Learning using Higher-Order Category Theory
Sridhar Mahadevan
We present a unified formalism for structure discovery of causal models and predictive state representation (PSR) models in reinforcement learning (RL) using higher-order category…
On The Universality of Diagrams for Causal Inference and The Causal Reproducing Property
Sridhar Mahadevan
We propose Universal Causality, an overarching framework based on category theory that defines the universal property that underlies causal inference independent of the underlying…