Neural Networks Fail to Learn Periodic Functions and How to Fix It

Abstract

Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a "periodic" inductive bias. As a fix of this problem, we propose a new activation, namely, $x+{\sin }^2(x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.

NeurIPS 2020 Camera Ready version

References in corpus (4)

• On the difficulty of training Recurrent Neural Networks
• Searching for Activation Functions
• Temperature dependence of COVID-19 transmission
• LaProp: Separating Momentum and Adaptivity in Adam

Cited by in corpus (6)

• Physical Activation Functions (PAFs): An Approach for More Efficient Induction of Physics into Physics-Informed Neural Networks (PINNs)
• An unsupervised latent/output physics-informed convolutional-LSTM network for solving partial differential equations using peridynamic differential operator
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
• Physics-informed neural network for acoustic resonance analysis in a one-dimensional acoustic tube
• Speech Enhancement Using Continuous Embeddings of Neural Audio Codec
• Can Synthetic Data Improve Symbolic Regression Extrapolation Performance?