Algorithm Development in Neural Networks: Insights from the Streaming Parity Task

Loek van Rossem, Andrew M. Saxe

Even when massively overparameterized, deep neural networks show a remarkable ability to generalize. Research on this phenomenon has focused on generalization within distribution,…