Depth Without the Magic: Inductive Bias of Natural Gradient Descent

Anna Kerekes, Anna Mészáros, Ferenc Huszár

In gradient descent, changing how we parametrize the model can lead to drastically different optimization trajectories, giving rise to a surprising range of meaningful inductive bi…