Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit

Abstract

Application-specific optical processors have been considered disruptive technologies for modern computing that can fundamentally accelerate the development of artificial intelligence (AI) by offering substantially improved computing performance. Recent advancements in optical neural network architectures for neural information processing have been applied to perform various machine learning tasks. However, the existing architectures have limited complexity and performance; and each of them requires its own dedicated design that cannot be reconfigured to switch between different neural network models for different applications after deployment. Here, we propose an optoelectronic reconfigurable computing paradigm by constructing a diffractive processing unit (DPU) that can efficiently support different neural networks and achieve a high model complexity with millions of neurons. It allocates almost all of its computational operations optically and achieves extremely high speed of data modulation and large-scale network parameter updating by dynamically programming optical modulators and photodetectors. We demonstrated the reconfiguration of the DPU to implement various diffractive feedforward and recurrent neural networks and developed a novel adaptive training approach to circumvent the system imperfections. We applied the trained networks for high-speed classifying of handwritten digit images and human action videos over benchmark datasets, and the experimental results revealed a comparable classification accuracy to the electronic computing approaches. Furthermore, our prototype system built with off-the-shelf optoelectronic components surpasses the performance of state-of-the-art graphics processing units (GPUs) by several times on computing speed and more than an order of magnitude on system energy efficiency.

References in corpus (3)

• Attojoule Optoelectronics for Low-Energy Information Processing and Communications: a Tutorial Review
• Human action recognition with a large-scale brain-inspired photonic computer
• A Scalable Photonic Computer Solving the Subset Sum Problem

Cited by in corpus (62)

• Single-chip photonic deep neural network for instantaneous image classification
• The physics of optical computing
• An optical neural network using less than 1 photon per multiplication
• Image sensing with multilayer, nonlinear optical neural networks
• Deep Learning with Coherent VCSEL Neural Networks
• Neuromorphic computing using wavelength-division multiplexing
• Prospects and applications of photonic neural networks
• Gigahertz free-space electro-optic modulators based on Mie resonances
• Optical coherent dot-product chip for sophisticated deep learning regression
• Polarization Multiplexed Diffractive Computing: All-Optical Implementation of a Group of Linear Transformations Through a Polarization-Encoded Diffractive Network
• On-demand harnessing of photonic soliton molecules
• Massively Parallel Universal Linear Transformations using a Wavelength-Multiplexed Diffractive Optical Network
• All-optical image classification through unknown random diffusers using a single-pixel diffractive network
• An Electro-Photonic System for Accelerating Deep Neural Networks
• Photonic real time video image signal processor at 17Tb/s based on a Kerr microcomb
• Unidirectional Imaging using Deep Learning-Designed Materials
• Optical Generative Models
• Deep Photonic Reservoir Computer Based on Frequency Multiplexing with Fully Analog Connection Between Layers
• Scaling optical computing in synthetic frequency dimension using integrated cavity acousto-optics
• Universal Polarization Transformations: Spatial programming of polarization scattering matrices using a deep learning-designed diffractive polarization transformer
• Classification and reconstruction of spatially overlapping phase images using diffractive optical networks
• Antiferromagnetic spatial photonic Ising machine through optoelectronic correlation computing
• Skydiver: A Spiking Neural Network Accelerator Exploiting Spatio-Temporal Workload Balance
• Photonic Extreme Learning Machine based on frequency multiplexing
• Simultaneously sorting vector vortex beams of 120 modes
• Ultrahigh-fidelity spatial mode quantum gates in high-dimensional space by diffractive deep neural networks
• Wavelength-division multiplexing optical Ising simulator enabling fully programmable spin couplings and external magnetic fields
• Energy-efficient neural network inference with microcavity exciton-polaritons
• Training neural networks with end-to-end optical backpropagation
• Quantum-limited stochastic optical neural networks operating at a few quanta per activation
• On-demand Photonic Ising Machine with Simplified Hamiltonian Calculation by Phase encoding and Intensity Detection
• Time-lapse image classification using a diffractive neural network
• Multiplexed all-optical permutation operations using a reconfigurable diffractive optical network
• Programmable metasurfaces for future photonic artificial intelligence
• Mechanical intelligence via fully reconfigurable elastic neuromorphic metasurfaces
• Emergent self-adaptation in an integrated photonic neural network for backpropagation-free learning
• Integration of Programmable Diffraction with Digital Neural Networks
• Asymmetrical estimator for training encapsulated deep photonic neural networks
• Two-color optically-addressed spatial light modulator as generic spatio-temporal systems
• Optical convolutional neural network with atomic nonlinearity
• 16-channel Photonic Solver for Optimization Problems on a Silicon Chip
• Multiplexed gradient descent: Fast online training of modern datasets on hardware neural networks without backpropagation
• Decision-making and control with diffractive optical networks
• Programmable photonic time circuits for highly scalable universal unitaries
• AnalogVNN: A fully modular framework for modeling and optimizing photonic neural networks
• High-performance real-world optical computing trained by in situ gradient-based model-free optimization
• Optoelectronic generative adversarial networks
• Artificial Nanophotonic Neuron with Internal Memory for Biologically Inspired and Reservoir Network Computing
• Optical Neural Ordinary Differential Equations
• Broadband Unidirectional Visible Imaging Using Wafer-Scale Nano-Fabrication of Multi-Layer Diffractive Optical Processors
• A 262 TOPS Hyperdimensional Photonic AI Accelerator powered by a Si3N4 microcomb laser
• SUANPAN: Scalable Photonic Linear Vector Machine
• A spiking photonic neural network of 40.000 neurons, trained with rank-order coding for leveraging sparsity
• Coherence Awareness in Diffractive Neural Networks
• Automating Experimental Optics with Sample Efficient Machine Learning Methods
• Quantum-secure multiparty deep learning
• Metalens array for complex-valued optical discrete Fourier transform
• Online training and pruning of multi-wavelength photonic neural networks
• Self-configuring high-speed multi-plane light conversion
• Building and aligning a 10-plane light converter
• All-Optical Doubly Resonant Cavities for ReLU Function in Nanophotonic Deep Learning
• Implementation of transformer-based LLMs with large-scale optoelectronic neurons on a CMOS compatible platform