Supervised sparse auto-encoders for interpretable and compositional representations

Ouns El Harzli, Hugo Wallner, Yoonsoo Nam +1

Sparse auto-encoders (SAEs) have re-emerged as a prominent method for mechanistic interpretability, yet they face two significant challenges: the non-smoothness of the $L_1$ penalt…