Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate

A. Bochkov

We study a constrained training regime for decoder-only Transformers in which the token interface is fixed, previously trained dense blocks are not reopened, and the active trainab…