The Persian Rug: solving toy models of superposition using large-scale symmetries

Aditya Cowsik, Kfir Dolev, Alex Infanger

We present a complete mechanistic description of the algorithm learned by a minimal non-linear sparse data autoencoder in the limit of large input dimension. The model, originally…