Attractor Patch Networks: Reducing Catastrophic Forgetting with Routed Low-Rank Patch Experts

Shashank

Transformers achieve strong language modeling accuracy, yet their position-wise feed-forward networks (FFNs) are dense, globally shared, and typically updated end to end. These pro…