HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization

Zhentao Tan, Wei Chen, Jingyi Shen +4

The quadratic complexity of attention poses a critical bottleneck for long-context processing, spurring interest in hybrid attention designs. Most open-source hybrid models adopt a…