ParaScopes: What do Language Models Activations Encode About Future Text?

Nicky Pochinkov, Yulia Volkova, Anna Vasileva +1

Interpretability studies in language models often investigate forward-looking representations of activations. However, as language models become capable of doing ever longer time h…