Gradient Descent as a Shrinkage Operator for Spectral Bias

Simon Lucey

We generalize the connection between activation function and spline regression/smoothing and characterize how this choice may influence spectral bias within a 1D shallow network. W…