Learning Rate Transfer for Hybrid Transformer-SSM Architectures

Jimin Seo, Gyubok Lee, Yeonsik Jo +11

We study learning rate (LR) scaling for hybrid architectures combining Transformer and State-Space Model (SSM) blocks, a class adopted by several recent production language models.…