Performance versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems

Abstract

We present the results of the comparative analysis of the performance versus complexity for several types of artificial neural networks (NNs) used for nonlinear channel equalization in coherent optical communication systems. The comparison has been carried out using an experimental set-up with transmission dominated by the Kerr nonlinearity and component imperfections. For the first time, we investigate the application to the channel equalization of the convolution layer (CNN) in combination with a bidirectional long short-term memory (biLSTM) layer and the design combining CNN with a multi-layer perceptron. Their performance is compared with the one delivered by the previously proposed NN equalizer models: one biLSTM layer, three-dense-layer perceptron, and the echo state network. Importantly, all architectures have been initially optimized by a Bayesian optimizer. We present the derivation of the computational complexity associated with each NN type -- in terms of real multiplications per symbol so that these results can be applied to a large number of communication systems. We demonstrated that in the specific considered experimental system the convolutional layer coupled with the biLSTM (CNN+biLSTM) provides the highest Q-factor improvement compared to the reference linear chromatic dispersion compensation (2.9 dB improvement). We examine the trade-off between the computational complexity and performance of all equalizers and demonstrate that the CNN+biLSTM is the best option when the computational complexity is not constrained, while when we restrict the complexity to lower levels, the three-layer perceptron provides the best performance. Our complexity analysis for different NNs is generic and can be applied in a wide range of physical and engineering systems.

References in corpus (10)

• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• LSTM: A Search Space Odyssey
• LSTM Fully Convolutional Networks for Time Series Classification
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Applying Neural Networks in Optical Communication Systems: Possible Pitfalls
• Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
• Compensation of Fiber Nonlinearities in Digital Coherent Systems Leveraging Long Short-Term Memory Neural Networks
• A Review of Designs and Applications of Echo State Networks
• Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere
• On the Statistical Challenges of Echo State Networks and Some Potential Remedies

Cited by in corpus (1)

• Transfer Learning for Neural Networks-based Equalizers in Coherent Optical Systems