Effects of sparsity and superposition on loss in simple autoencoders

Mriganka Basu Roy Chowdhury, Eric McLaughlin Weiner

One of the major difficulties in the mechanistic interpretability of neural networks is the occurrence of polysemanticity, which suggests that each neuron is typically responsible…