Understanding sparse autoencoder scaling in the presence of feature manifolds

Eric J. Michaud, Liv Gorton, Tom McGrath

Sparse autoencoders (SAEs) model the activations of a neural network as linear combinations of sparsely occurring directions of variation (latents). The ability of SAEs to reconstr…