Do Language Models Use Their Depth Efficiently?

Róbert Csordás, Christopher D. Manning, Christopher Potts

Modern LLMs are increasingly deep, and depth correlates with performance, albeit with diminishing returns. However, do these models use their depth efficiently? Do they compose mor…