Blind Equalization and Channel Estimation in Coherent Optical Communications Using Variational Autoencoders

Abstract

We investigate the potential of adaptive blind equalizers based on variational inference for carrier recovery in optical communications. These equalizers are based on a low-complexity approximation of maximum likelihood channel estimation. We generalize the concept of variational autoencoder (VAE) equalizers to higher order modulation formats encompassing probabilistic constellation shaping (PCS), ubiquitous in optical communications, oversampling at the receiver, and dual-polarization transmission. Besides black-box equalizers based on convolutional neural networks, we propose a model-based equalizer based on a linear butterfly filter and train the filter coefficients using the variational inference paradigm. As a byproduct, the VAE also provides a reliable channel estimation. We analyze the VAE in terms of performance and flexibility over a classical additive white Gaussian noise (AWGN) channel with inter-symbol interference (ISI) and over a dispersive linear optical dual-polarization channel. We show that it can extend the application range of blind adaptive equalizers by outperforming the state-of-the-art constant-modulus algorithm (CMA) for PCS for both fixed but also time-varying channels. The evaluation is accompanied with a hyperparameter analysis.

Published (Open Access) in IEEE Journal on Selected Areas in Communications, Sep 2022

References in corpus (1)

• Geometric Constellation Shaping for Phase-noise Channels Using a Differentiable Blind Phase Search

Cited by in corpus (4)

• End-to-end Optimization of Constellation Shaping for Wiener Phase Noise Channels with a Differentiable Blind Phase Search
• Joint PMD Tracking and Nonlinearity Compensation with Deep Neural Networks
• Machine Learning for Relaying Topology: Optimization of IoT Network with Energy Harvesting
• Model Order Selection with Variational Autoencoding