Characterizing Verbatim Short-Term Memory in Neural Language Models

Kristijan Armeni, Christopher Honey, Tal Linzen

When a language model is trained to predict natural language sequences, its prediction at each moment depends on a representation of prior context. What kind of information about t…