6 citations · 18 across the 13 of their papers we have counts for
6 papers · 1 filter
Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weights
Yichuan Deng, Zhao Song, Shenghao Xie +1
In the realm of deep learning, transformers have emerged as a dominant architecture, particularly in natural language processing tasks. However, with their widespread adoption, con…
Why Softmax Attention Outperforms Linear Attention
Yichuan Deng, Zhao Song, Kaijun Yuan +1
Large transformer models have achieved state-of-the-art results in numerous natural language processing tasks. Among the pivotal components of the transformer architecture, the att…
Attention Scheme Inspired Softmax Regression
Yichuan Deng, Zhihang Li, Zhao Song
Large language models (LLMs) have made transformed changes for human society. One of the key computation in LLMs is the softmax unit. This operation is important in LLMs because it…
Solving Tensor Low Cycle Rank Approximation
Yichuan Deng, Yeqi Gao, Zhao Song
Large language models have become ubiquitous in modern life, finding applications in various domains such as natural language processing, language translation, and speech recogniti…
Randomized and Deterministic Attention Sparsification Algorithms for Over-parameterized Feature Dimension
Yichuan Deng, Sridhar Mahadevan, Zhao Song
Large language models (LLMs) have shown their power in different areas. Attention computation, as an important subroutine of LLMs, has also attracted interests in theory. Recently…
An Improved Sample Complexity for Rank-1 Matrix Sensing
Yichuan Deng, Zhihang Li, Zhao Song
Matrix sensing is a problem in signal processing and machine learning that involves recovering a low-rank matrix from a set of linear measurements. The goal is to reconstruct the o…