When Does Re-initialization Work?

Sheheryar Zaidi, Tudor Berariu, Hyunjik Kim +4

Re-initializing a neural network during training has been observed to improve generalization in recent works. Yet it is neither widely adopted in deep learning practice nor is it o…