Memory Attention

Jiale Kang

Language models typically construct attention values from contextual hidden states, even when some of their content may be reusable across contexts. We investigate whether token-in…