9 citations · 80 across the 35 of their papers we have counts for
35 papers
The Fine-Grained Complexity of Gradient Computation for Training Large Language Models
Josh Alman, Zhao Song
Large language models (LLMs) have made fundamental contributions over the last a few years. To train an LLM, one needs to alternatingly run `forward' computations and `backward' co…
Fast Heavy Inner Product Identification Between Weights and Inputs in Neural Network Training
Lianke Qin, Saayan Mitra, Zhao Song +2
In this paper, we consider a heavy inner product identification problem, which generalizes the Light Bulb problem~(\cite{prr89}): Given two sets and $B \sub…
Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weights
Yichuan Deng, Zhao Song, Shenghao Xie +1
In the realm of deep learning, transformers have emerged as a dominant architecture, particularly in natural language processing tasks. However, with their widespread adoption, con…
How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computation
Josh Alman, Zhao Song
In the classical transformer attention scheme, we are given three size matrices (the query, key, and value tokens), and the goal is to compute a new $n \time…
Fine-tune Language Models to Approximate Unbiased In-context Learning
Timothy Chu, Zhao Song, Chiwun Yang
In-context learning (ICL) is an astonishing emergent ability of large language models (LLMs). By presenting a prompt that includes multiple input-output pairs as examples and intro…
A Unified Scheme of ResNet and Softmax
Zhao Song, Weixin Wang, Junze Yin
Large language models (LLMs) have brought significant changes to human society. Softmax regression and residual neural networks (ResNet) are two important techniques in deep learni…