5 papers · 1 filter
Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding
Themistoklis Haris, Henry Li, Maryam Karimzadehgan
Massive KV caches can cause severe memory-bandwidth bottlenecks during long-context decoding. Sparse attention methods mitigate this problem, but often drop necessary context, lead…
Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classification
Patrick Lutz, Themistoklis Haris, Arjun Chandra +2
Transformers can perform in-context classification from a few labeled examples, yet the inference-time algorithm remains opaque. We study multi-class linear classification in the h…
Noise Stability of Transformer Models
Themistoklis Haris, Zihan Zhang, Yuichi Yoshida
Understanding simplicity biases in deep learning offers a promising path toward developing reliable AI. A common metric for this, inspired by Boolean function analysis, is average…
Fast-MWEM: Private Data Release in Sublinear Time
Themistoklis Haris, Steve Choi, Mutiraj Laksanawisit
The Multiplicative Weights Exponential Mechanism (MWEM) is a fundamental iterative framework for private data analysis, with broad applications such as answering linear queries…
NN Attention Demystified: A Theoretical Exploration for Scalable Transformers
Themistoklis Haris
Despite their power, Transformers face challenges with long sequences due to the quadratic complexity of self-attention. To address this limitation, methods like -Nearest-Neighb…