3 citations · 10 across the 16 of their papers we have counts for
6 papers · 1 filter
A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformer
Chenyang Li, Zhao Song, Weixin Wang +1
The Deep Leakage from Gradient (DLG) attack has emerged as a prevalent and highly effective method for extracting sensitive training data by inspecting exchanged gradients. This ap…
One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Space
Raghav Addanki, Chenyang Li, Zhao Song +1
Attention computation takes both the time complexity of and the space complexity of simultaneously, which makes deploying Large Language Models (LLMs) in streamin…
Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weights
Yichuan Deng, Zhao Song, Shenghao Xie +1
In the realm of deep learning, transformers have emerged as a dominant architecture, particularly in natural language processing tasks. However, with their widespread adoption, con…
An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descent
Zhao Song, Chiwun Yang
The delta-bar-delta algorithm is recognized as a learning rate adaptation technique that enhances the convergence speed of the training process in optimization by dynamically sched…
Fine-tune Language Models to Approximate Unbiased In-context Learning
Timothy Chu, Zhao Song, Chiwun Yang
In-context learning (ICL) is an astonishing emergent ability of large language models (LLMs). By presenting a prompt that includes multiple input-output pairs as examples and intro…
How to Protect Copyright Data in Optimization of Large Language Models?
Timothy Chu, Zhao Song, Chiwun Yang
Large language models (LLMs) and generative AI have played a transformative role in computer research and applications. Controversy has arisen as to whether these models output cop…