Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviors

Henrik Klagges, Robert Dahlke, Fabian Klemm +4

Requiring $10^{13}$-$10^{15}$ FLOPs to calculate one 8 bit weight in an LLM during pretraining is extremely expensive and seems inefficient. To better leverage the huge investments…