Rational Neural Networks have Expressivity Advantages

Maosen Tang, Alex Townsend

We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smo…