Backpropagation through nonlinear units for all-optical training of neural networks

Abstract

Backpropagation through nonlinear neurons is an outstanding challenge to the field of optical neural networks and the major conceptual barrier to all-optical training schemes. Each neuron is required to exhibit a directionally dependent response to propagating optical signals, with the backwards response conditioned on the forward signal, which is highly non-trivial to implement optically. We propose a practical and surprisingly simple solution that uses saturable absorption to provide the network nonlinearity. We find that the backward propagating gradients required to train the network can be approximated in a pump-probe scheme that requires only passive optical elements. Simulations show that, with readily obtainable optical depths, our approach can achieve equivalent performance to state-of-the-art computational networks on image classification benchmarks, even in deep networks with multiple sequential gradient approximations. This scheme is compatible with leading optical neural network proposals and therefore provides a feasible path towards end-to-end optical training.

arXiv commentError fixed in Fig.1

References in corpus (6)

• Neural Message Passing for Quantum Chemistry
• All-Optical Machine Learning Using Diffractive Deep Neural Networks
• Many-body quantum state tomography with neural networks
• All Optical Neural Network with Nonlinear Activation Functions
• Reinforcement Learning in a large scale photonic Recurrent Neural Network
• On-Chip Optical Convolutional Neural Networks

Cited by in corpus (2)

• Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment
• Light-in-the-loop: using a photonics co-processor for scalable training of neural networks