An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars

Vinoth Nandakumar, Qiang Qu, Pramod Thebe +2

Deep neural networks are widely believed to derive their expressive power from their ability to form \textbf{hierarchical representations}, capturing progressively more abstract an…