3 citations · 10 across the 16 of their papers we have counts for
5 papers · 1 filter
Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
Yekun Ke, Yingyu Liang, Zhenmei Shi +2
The application of transformer-based models on time series forecasting (TSF) tasks has long been popular to study. However, many of these works fail to beat the simple linear resid…
Unlocking the Theory Behind Scaling 1-Bit Neural Networks
Majid Daliri, Zhao Song, Chiwun Yang
Recently, 1-bit Large Language Models (LLMs) have emerged, showcasing an impressive combination of efficiency and performance that rivals traditional LLMs. Research by Wang et al.…
Towards Infinite-Long Prefix in Transformer
Yingyu Liang, Zhenmei Shi, Zhao Song +1
Prompting and context-based fine-tuning methods, which we call Prefix Learning, have been proposed to enhance the performance of language models on various downstream tasks. They a…
How Sparse Attention Approximates Exact Attention? Your Attention is Naturally -Sparse
Yichuan Deng, Zhao Song, Jing Xiong +1
Sparse Attention is a technique that approximates standard attention computation with sub-quadratic complexity. This is achieved by selectively ignoring smaller entries in the atte…
Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence
Yichuan Deng, Zhao Song, Chiwun Yang
Based on SGD, previous works have proposed many algorithms that have improved convergence speed and generalization in stochastic optimization, such as SGDm, AdaGrad, Adam, etc. How…