3 citations · 5 across the 7 of their papers we have counts for
7 papers
When Can We Solve the Weighted Low Rank Approximation Problem in Truly Subquadratic Time?
Chenyang Li, Yingyu Liang, Zhenmei Shi +1
The weighted low-rank approximation problem is a fundamental numerical linear algebra problem and has many applications in machine learning. Given a weight matrix …
Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency
Jerry Yao-Chieh Hu, Wei-Po Wang, Ammar Gilani +3
We investigate the statistical and computational limits of prompt tuning for transformer-based foundation models. Our key contributions are prompt tuning on \emph{single-head} tran…
Inverting the Leverage Score Gradient: An Efficient Approximate Newton Method
Chenyang Li, Zhao Song, Zhaoxing Xu +1
Leverage scores have become essential in statistics and machine learning, aiding regression analysis, randomized matrix computations, and various other tasks. This paper delves int…
Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyond
Yang Cao, Yingyu Liang, Zhenmei Shi +1
The softmax activation function plays a crucial role in the success of large language models (LLMs), particularly in the self-attention mechanism of the widely adopted Transformer…
Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs
Chenyang Li, Yingyu Liang, Zhenmei Shi +2
In the evolving landscape of machine learning, a pivotal challenge lies in deciphering the internal representations harnessed by neural networks and Transformers. Building on recen…
A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformer
Chenyang Li, Zhao Song, Weixin Wang +1
The Deep Leakage from Gradient (DLG) attack has emerged as a prevalent and highly effective method for extracting sensitive training data by inspecting exchanged gradients. This ap…
One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Space
Raghav Addanki, Chenyang Li, Zhao Song +1
Attention computation takes both the time complexity of and the space complexity of simultaneously, which makes deploying Large Language Models (LLMs) in streamin…