Knee-Deep in C-RASP: A Transformer Depth Hierarchy

Andy Yang, Michaël Cadilhac, David Chiang

It has been observed that transformers with greater depth (that is, more layers) have more capabilities, but can we establish formally which capabilities are gained? We answer this…