7 papers
Training Tensor Attention Efficiently: From Cubic to Almost Linear Time
Yang Cao, Yingyu Liang, Zhenmei Shi +1
Tensor Attention, a multi-view attention that is able to capture high-order correlations among multiple modalities, can overcome the representational limitations of classical matri…
Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyond
Yang Cao, Yingyu Liang, Zhenmei Shi +1
The softmax activation function plays a crucial role in the success of large language models (LLMs), particularly in the self-attention mechanism of the widely adopted Transformer…
Provable Differentially Private Computation of the Cross-Attention Mechanism
Yekun Ke, Yingyu Liang, Zhenmei Shi +2
Cross-attention has emerged as a cornerstone module in modern artificial intelligence, underpinning critical applications such as retrieval-augmented generation (RAG), system promp…
Fast John Ellipsoid Computation with Differential Privacy Optimization
Xiaoyu Li, Yingyu Liang, Zhenmei Shi +2
Determining the John ellipsoid - the largest volume ellipsoid contained within a convex polytope - is a fundamental problem with applications in machine learning, optimization, and…
Differential Privacy Mechanisms in Neural Tangent Kernel Regression
Jiuxiang Gu, Yingyu Liang, Zhizhou Sha +2
Training data privacy is a fundamental problem in modern Artificial Intelligence (AI) applications, such as face recognition, recommendation systems, language generation, and many…
Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers
Yingyu Liang, Heshan Liu, Zhenmei Shi +3
The self-attention mechanism is the key to the success of transformers in recent Large Language Models (LLMs). However, the quadratic computational cost in the input seque…