11 TeraFLOPs per second photonic convolutional accelerator for deep learning optical neural networks

Abstract

Convolutional neural networks (CNNs), inspired by biological visual cortex systems, are a powerful category of artificial neural networks that can extract the hierarchical features of raw data to greatly reduce the network parametric complexity and enhance the predicting accuracy. They are of significant interest for machine learning tasks such as computer vision, speech recognition, playing board games and medical diagnosis. Optical neural networks offer the promise of dramatically accelerating computing speed to overcome the inherent bandwidth bottleneck of electronics. Here, we demonstrate a universal optical vector convolutional accelerator operating beyond 10 TeraFLOPS (floating point operations per second), generating convolutions of images of 250,000 pixels with 8 bit resolution for 10 kernels simultaneously, enough for facial image recognition. We then use the same hardware to sequentially form a deep optical CNN with ten output neurons, achieving successful recognition of full 10 digits with 900 pixel handwritten digit images with 88% accuracy. Our results are based on simultaneously interleaving temporal, wavelength and spatial dimensions enabled by an integrated microcomb source. This approach is scalable and trainable to much more complex networks for demanding applications such as unmanned vehicle and real-time video recognition.

21 pages, 9 figures, 39 references

References in corpus (12)

• Deep Learning with Coherent Nanophotonic Circuits
• All-Optical Machine Learning Using Diffractive Deep Neural Networks
• All-optical signal processing platforms for CMOS compatible integrated nonlinear optics
• Microresonator solitons for massively parallel coherent optical communications
• An Integrated-Photonics Optical-Frequency Synthesizer
• Fully integrated ultra-low power Kerr comb generation
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• Integrated micro-comb sources for quantum optical applications
• Attojoule Optoelectronics for Low-Energy Information Processing and Communications: a Tutorial Review
• Harnessing optical micro-combs for microwave photonics
• Optical combs with a crystalline whispering gallery mode resonator
• Soliton crystals in Kerr resonators

Cited by in corpus (133)

• Single-chip photonic deep neural network for instantaneous image classification
• Graphene oxide: new opportunities for optoelectronic, electronic and photonic chips
• High performance photonic microwave filters based on a 50GHz optical soliton crystal Kerr micro-comb
• Bridging microcombs and silicon photonic engines for optoelectronics systems
• High-yield wafer-scale fabrication of ultralow-loss, dispersion-engineered silicon nitride photonic circuits
• The physics of optical computing
• On scientific understanding with artificial intelligence
• High bandwidth temporal RF photonic signal processing with Kerr micro-combs: integration, fractional differentiation and Hilbert transforms
• Applications of integrated optical microcombs
• Intelligent Computing: The Latest Advances, Challenges and Future
• An optical neural network using less than 1 photon per multiplication
• Image sensing with multilayer, nonlinear optical neural networks
• Highly tunable broadband RF photonic fractional Hilbert transformer based on a Kerr soliton crystal microcomb source
• High-efficiency and broadband electro-optic frequency combs enabled by coupled micro-resonators
• Integral order photonic RF and microwave signal processors based on soliton crystal Kerr micro-combs
• Neuromorphic computing using wavelength-division multiplexing
• Deep Learning with Coherent VCSEL Neural Networks
• Single chip photonic deep neural network with accelerated training
• Prospects and applications of photonic neural networks
• Photonic Microwave and RF Channelizers using Kerr Micro-combs
• High-order tensor flow processing using integrated photonic circuits
• Sagnac interference in integrated photonics for reflection mirrors, gyroscopes, filters, and wavelength interleavers
• In-memory photonic dot-product engine with electrically programmable weight banks
• Frequency comb distillation for optical superchannel transmission
• Third-order nonlinear optical response of 2D materials in the telecom band
• Platicon microcomb generation using laser self-injection locking
• Optical coherent dot-product chip for sophisticated deep learning regression
• Polarization Multiplexed Diffractive Computing: All-Optical Implementation of a Group of Linear Transformations Through a Polarization-Encoded Diffractive Network
• Photonics for Neuromorphic Computing: Fundamentals, Devices, and Opportunities
• Theory for the Accuracy of Microcomb Photonic Microwave Transversal Signal Processors
• Photonic extreme learning machine by free-space optical propagation
• Nonlinear optical encoding enabled by recurrent linear scattering
• Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment
• Spectral Shaping with Integrated Self-Coupled Sagnac Loop Reflectors
• Massively Parallel Universal Linear Transformations using a Wavelength-Multiplexed Diffractive Optical Network
• Reservoir computing based on a silicon microring and time multiplexing for binary and analog operations
• Nonlinear co-generation of graphene plasmons for optoelectronic logic operations
• Photo-thermal tuning of graphene oxide coated integrated optical waveguides
• Synthetic-reflection self-injection-locked microcombs
• Asymptotically Fault-Tolerant Programmable Photonics
• Soliton crystal Kerr microcombs for high-speed, scalable optical neural networks at 10 GigaOPs/s
• An Electro-Photonic System for Accelerating Deep Neural Networks
• Optical data transmission field trial @ 44Tb/s with a 49GHz Kerr soliton crystal microcomb
• Ultra-high bandwidth fiber-optic data transmission with a single chip source
• Octave-spanning Kerr soliton frequency combs in dispersion- and dissipation-engineered lithium niobate microresonators
• Photonic neural networks based on integrated silicon microresonators
• High-speed tunable microwave-rate soliton microcomb
• Photonic real time video image signal processor at 17Tb/s based on a Kerr microcomb
• Dispersion-less Kerr solitons in spectrally confined optical cavities
• Optical Kerr soliton microcombs for high bandwidth communications
• Maximizing the performance for microcomb based microwave photonic transversal signal processors
• Cascadable all-optical NAND gates using diffractive networks
• Optical Generative Models
• Deep Photonic Reservoir Computer Based on Frequency Multiplexing with Fully Analog Connection Between Layers
• Thermal control of Kerr microresonator soliton comb via an optical sideband
• Designing and analyzing microwave photonic spectral domain filters based on transversal filtering with optical microcombs
• Scaling optical computing in synthetic frequency dimension using integrated cavity acousto-optics
• Photonic Convolution Neural Network Based on Interleaved Time-Wavelength Modulation
• Waveguide optical parametric amplifiers in silicon nitride with 2D graphene oxide films
• Scalability of all-optical neural networks based on spatial light modulators
• Photonic Extreme Learning Machine based on frequency multiplexing
• Inverse Design of Grating Couplers Using the Policy Gradient Method from Reinforcement Learning
• Temporal Walk-off Induced Dissipative Quadratic Solitons
• Hardware realization of the multiply and accumulate operation on radio-frequency signals with magnetic tunnel junctions
• Photonic reservoir computer based on frequency multiplexing
• An array of microresonators as a Photonic Extreme Learning Machine
• Thermal investigation of bistability in high index doped silica integrated ring resonators
• Energy-efficient neural network inference with microcavity exciton-polaritons
• A photonic complex perceptron for ultrafast data processing
• Training neural networks with end-to-end optical backpropagation
• Complex-valued universal linear transformations and image encryption using spatially incoherent diffractive networks
• Quantum-limited stochastic optical neural networks operating at a few quanta per activation
• Coherent terahertz radiation with 2.8-octave tunability through chip-scale photomixed microresonator optical parametric oscillation
• Design of microring resonators integrated with 2D graphene oxide films for four-wave mixing
• Ultrafast single-channel machine vision based on neuro-inspired photonic computing
• Vibrational Kerr solitons in an optomechanical microresonator
• High Performance Optical Filters Using Three Waveguide Coupled Sagnac Loop Reflectors
• Spectral engineering of optical microresonators in anisotropic lithium niobate crystal
• Supercontinuum Generation by Saturated Second-Order Nonlinear Interactions
• Programmable access to microresonator solitons with modulational sideband heating
• Near octave-spanning perfect soliton crystals in AlN microresonators
• Optical microcombs in whispering gallery mode crystalline resonators with dispersive intermode interactions
• Microwave photonic transversal filters based on microcombs with feedback control
• The performance of microwave photonic signal processors based on microcombs with different input signal waveforms
• Programmable metasurfaces for future photonic artificial intelligence
• Optical Neural Network Based on Synthetic Nonlinear Photonic Lattices
• Transferable polychromatic optical encoder for neural networks
• All-Optical Image Identification with Programmable Matrix Transformation
• Contrasting the relative performance of RF photonic transversal signal processors based on microcombs using discrete components versus integrated devices
• Deep Photonic Reservoir Computer for Speech Recognition
• Waveguide-multiplexed photonic matrix-vector multiplication processor using multiport photodetectors
• Mirage: An RNS-Based Photonic Accelerator for DNN Training
• Integration of Programmable Diffraction with Digital Neural Networks
• Integrated and DC-powered superconducting microcomb
• Hardware-Efficient Photonic Tensor Core: Accelerating Deep Neural Networks with Structured Compression
• A Blueprint for Precise and Fault-Tolerant Analog Neural Networks
• AnalogVNN: A fully modular framework for modeling and optimizing photonic neural networks
• Neural networks based on ultrafast time-delayed effects in exciton-polaritons
• Single-Shot Matrix-Matrix Multiplication Optical Tensor Processor for Deep Learning
• Optical neural network architecture for deep learning with the temporal synthetic dimension
• Optical neuromorphic computing via temporal up-sampling and trainable encoding on a telecom device platform
• High-speed photon correlation monitoring of amplified quantum noise by chaos using deep-learning balanced homodyne detection
• Inverse design of plasma metamaterial devices with realistic elements
• Solving multi-armed bandit problems using a chaotic microresonator comb
• Graphene oxide for high performance nonlinear optics in integrated nanowires
• Temporal dissipative structures in optical Kerr resonators with transient loss fluctuation
• A 262 TOPS Hyperdimensional Photonic AI Accelerator powered by a Si3N4 microcomb laser
• Graphene oxide 2D films integrated with nanowires and ring resonators for enhanced nonlinear optics
• Photonic Neuromorphic Accelerators for Event-Based Imaging Flow Cytometry
• High 3rd Order Optical Kerr Nonlinearity of PdSe2 Di-chalcogenide 2D Films for Nonlinear Photonic Chips
• Photonic single perceptron at Giga-OP/s speeds with Kerr microcombs for scalable optical neural networks
• Photonic neural field on a silicon chip: large-scale, high-speed neuro-inspired computing and sensing
• Self-cooling, blue-detuned dissipative Kerr microresonator soliton comb
• Partial Differential Equations is All You Need for Generating Neural Architectures -- A Theory for Physical Artificial Intelligence Systems
• Temporal bandwidth of consecutive polariton condensation
• The Impact of Feature Representation on the Accuracy of Photonic Neural Networks
• Harnessing Optoelectronic Noises in a Photonic Generative Network
• Spatial Optical Simulator for Classical Statistical Models
• AI based design of 2D material integrated optical polarizers
• Brillouin-based storage of QPSK signals with fully tunable phase retrieval
• Quantum-secure multiparty deep learning
• Diffractive Interconnects: All-Optical Permutation Operation Using Diffractive Networks
• Photonic Differential Privacy with Direct Feedback Alignment
• Perspective of high-speed Mach-Zehnder modulators based on nonlinear optics and complex band structures
• An experimental demonstration of neuromorphic sensing of chemical species using electro-optical reservoir computing
• The role of the inverse Cherenkov effect in the formation of ultrashort Raman solitons in silica microspheres
• Slimmed optical neural networks with multiplexed neuron sets and a corresponding backpropagation training algorithm
• Mapping ultrafast timing jitter in dispersion-managed 89 GHz frequency microcombs via self-heterodyne linear interferometry
• Emergence of biased errors in imperfect photonic circuits
• 2022 Roadmap on Neuromorphic Computing and Engineering
• Implementation of transformer-based LLMs with large-scale optoelectronic neurons on a CMOS compatible platform
• Improvement of image classification by multiple optical scattering
• All-Optical Doubly Resonant Cavities for ReLU Function in Nanophotonic Deep Learning