Sky sphere representation in language models

Aleksandr Berdnikov, Yevgeny Liokumovich

We analyze whether language models of size ~100B have a representation of the night sky map that is decodable from their residual stream. We find that most of the considered open-s…