Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment

Abstract

The scaling hypothesis motivates the expansion of models past trillions of parameters as a path towards better performance. Recent significant developments, such as GPT-3, have been driven by this conjecture. However, as models scale-up, training them efficiently with backpropagation becomes difficult. Because model, pipeline, and data parallelism distribute parameters and gradients over compute nodes, communication is challenging to orchestrate: this is a bottleneck to further scaling. In this work, we argue that alternative training methods can mitigate these issues, and can inform the design of extreme-scale training hardware. Indeed, using a synaptically asymmetric method with a parallelizable backward pass, such as Direct Feedback Alignement, communication needs are drastically reduced. We present a photonic accelerator for Direct Feedback Alignment, able to compute random projections with trillions of parameters. We demonstrate our system on benchmark tasks, using both fully-connected and graph convolutional networks. Our hardware is the first architecture-agnostic photonic co-processor for training neural networks. This is a significant step towards building scalable hardware, able to go beyond backpropagation, and opening new avenues for deep learning.

6 pages, 2 figures, 1 table. Oral at the Beyond Backpropagation Workshop, NeurIPS 2020

References in corpus (7)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Language Models are Few-Shot Learners
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Training Neural Networks with Local Error Signals
• Random Projections through multiple optical scattering: Approximating kernels at the speed of light
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• Backpropagation through nonlinear units for all-optical training of neural networks

Cited by in corpus (4)

• Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment
• Self-Contrastive Forward-Forward Algorithm
• Model-free Optical Processors using In Situ Reinforcement Learning with Proximal Policy Optimization
• Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms