4 papers
The Role of Feed-Forward Layers in Transformer Dynamics
Thomas Jacob Maranzatto, Semih Akkoc, Sennur Ulukus
We study the dynamical behavior of tokens in transformers from a control-theoretic perspective. Our model includes the feed-forward layer present after the self-attention mechanism…
The Hidden Perception Constraint in Task-Aware Compression
Sahan Liyanaarachchi, Semih Akkoc, Sennur Ulukus +1
With the recent advancements of neural compressors, explicitly incorporating perception constraints into the design of compression schemes has gained significant attention. Traditi…
On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers
Sixu Li, Thomas Jacob Maranzatto, Jan Peszek +5
We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems. In this perspective, tokens are modeled as particles…
The Rate-Distortion-Deception Tradeoff
Semih Akkoc, Sahan Liyanaarachchi, Sennur Ulukus +1
The problem of finding the optimal compression rate for a given random variable has been traditionally studied under two main constraints: distortion and perception. The distortion…