On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders

Elana Simon, Etowah Adams, James Zou

Sparse autoencoders (SAEs) decompose neural network activations into interpretable features, but many learned features never activate, a problem called feature death that wastes di…