When Does Sparsity Mitigate the Curse of Depth in LLMs

Dilxat Muhtar, Xinyuan Song, Sebastian Pokutta +4

Recent work has demonstrated the curse of depth in large language models (LLMs), where later layers contribute less to learning and representation than earlier layers. Such under-u…