Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory

Daniel Goldstein, Navneel Singhal, Eugene Cheah

Recall presents a difficult choice: transformers have a linearly growing memory that slows each successive token, while linear RNNs typically have fixed costs but limited recall. W…