Initializing Models with Larger Ones

Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov +5

Weight initialization plays an important role in neural network training. Widely used initialization methods are proposed and evaluated for networks that are trained from scratch.…