Short Horizons and Sparse Concepts: a Mathematical View of the Readout in the J-lens

Shi-Qi Yan, Kai-Xuan Ding, Chao-Hong Tan +4

The Jacobian lens (J-lens) has been proposed as a way to read verbalizable representations from language models. However, its principle and meaning lack a detailed and theoretical…