6 papers
On the suboptimality of linear codes for binary distributed hypothesis testing
Adway Girish, Robinson D. H. Cung, Emre Telatar
We study a binary distributed hypothesis testing problem where two agents observe correlated binary vectors and communicate compressed information at the same rate to a central dec…
High signal-to-noise ratio asymptotics of entropy-constrained Gaussian channel capacity
Adway Girish, Shlomo Shamai, Emre Telatar
We study the input-entropy-constrained Gaussian channel capacity problem in the asymptotic high signal-to-noise ratio (SNR) regime. We show that the capacity-achieving distribution…
Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains
Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish +4
Attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages. To deepen our understanding of their sequential modeling…
On entropy-constrained Gaussian channel capacity via the moment problem
Adway Girish, Shlomo Shamai, Emre Telatar
We study the capacity of the power-constrained additive Gaussian channel with an entropy constraint at the input. In particular, we characterize this capacity in the low signal-to-…
Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
Alliot Nagle, Adway Girish, Marco Bondaschi +3
We formalize the problem of prompt compression for large language models (LLMs) and present a framework to unify token-level prompt compression methods which create hard prompts fo…
Local to Global: Learning Dynamics and Effect of Initialization for Transformers
Ashok Vardhan Makkuva, Marco Bondaschi, Chanakya Ekbote +4
In recent years, transformer-based models have revolutionized deep learning, particularly in sequence modeling. To better understand this phenomenon, there is a growing interest in…