Towards interpreting any residual stream direction by scaling activation inversion

Celeste De Schamphelaere, Juan J. Vazquez, Ari Spiesberger +4

To interpret a direction in a language model's residual stream, we can look for text that strongly activates it. But when such text is found by scanning a corpus, it is limited to…