Beyond Gaussian Initializations: Signal Preserving Weight Initialization for Odd-Sigmoid Activations

Hyunwoo Lee, Hayoung Choi, Hyunju Kim

Activation functions critically influence trainability and expressivity, and recent work has therefore explored a broad range of nonlinearities. However, widely used Gaussian i.i.d…