2 citations · 4 across the 20 of their papers we have counts for
5 papers · 1 filter
SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers
Aref Jafari, Yuhe Fan, Benyamin Jamialahmadi +3
Transformers have demonstrated strong performance across a wide range of sequence modeling tasks, but their quadratic attention complexity limits scalability to long sequences. Lin…
DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers
Aman Sharma, Saeed Najafi, Parsa Farinneya +6
Transformers achieve state-of-the-art results across many tasks, but their uniform application of quadratic self-attention to every token at every layer makes them computationally…
Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models
Benyamin Jamialahmadi, Parsa Kavehzadeh, Mehdi Rezagholizadeh +5
Deploying large language models (LLMs) in real-world applications is often hindered by strict computational and latency constraints. While dynamic inference offers the flexibility…
ReGLA: Refining Gated Linear Attention
Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh +2
Recent advancements in Large Language Models (LLMs) have set themselves apart with their exceptional performance in complex language modelling tasks. However, these models are also…
ZETA: Leveraging Z-order Curves for Efficient Top-k Attention
Qiuhao Zeng, Jerry Huang, Peng Lu +4
Over recent years, the Transformer has become a fundamental building block for sequence modeling architectures. Yet at its core is the use of self-attention, whose memory and compu…