On the expressivity of deep Heaviside networks

Insung Kong, Juntong Chen, Sophie Langer +1

We show that deep Heaviside networks (DHNs) have limited expressiveness but that this can be overcome by including either skip connections or neurons with linear activation. We pro…