Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes

Zhenfeng Tu, Santiago Aranguri, Arthur Jacot

The training dynamics of linear networks are well studied in two distinct setups: the lazy regime and balanced/active regime, depending on the initialization and width of the netwo…