Mixture-of-Depths Attention

Lianghui Zhu, Yuxin Fang, Bencheng Liao +10

Scaling depth is a key driver for large language models (LLMs). Yet, as LLMs become deeper, they often suffer from signal degradation: informative features formed in shallow layers…