Activation Transport Operators

Andrzej Szablewski, Marek Masiak

The residual stream mediates communication between transformer decoder layers via linear reads and writes of non-linear computations. While sparse-dictionary learning-based methods…