Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale

James A. Michaelov, Roger P. Levy, Benjamin K. Bergen

We show that across architecture (Transformer vs. Mamba vs. RWKV), training dataset (OpenWebText vs. The Pile), and scale (14 million parameters to 12 billion parameters), autoregr…