The Effect of Layer Ordering in Hybrid Architectures

John Cooper, Mingchen Ma, Jiaqi Mo +4

Hybrid attention is rapidly becoming the dominant architectural choice for frontier models. Despite their successes, such architectures complicate model design by introducing addit…