Breaking the Block: Preserving Data Continuity to Train Superior SAEs for Instruct Models

Jiaming Li, Haoran Ye, Yukun Chen +5

Sparse Autoencoders (SAEs) are a cornerstone of mechanistic interpretability. Existing training methods inherit the Block Training paradigm from LLM pre-training, which introduces…