Parallel convolution processing using an integrated photonic tensor core

Abstract

With the proliferation of ultra-high-speed mobile networks and internet-connected devices, along with the rise of artificial intelligence, the world is generating exponentially increasing amounts of data - data that needs to be processed in a fast, efficient and smart way. These developments are pushing the limits of existing computing paradigms, and highly parallelized, fast and scalable hardware concepts are becoming progressively more important. Here, we demonstrate a computational specific integrated photonic tensor core - the optical analog of an ASIC-capable of operating at Tera-Multiply-Accumulate per second (TMAC/s) speeds. The photonic core achieves parallelized photonic in-memory computing using phase-change memory arrays and photonic chip-based optical frequency combs (soliton microcombs). The computation is reduced to measuring the optical transmission of reconfigurable and non-resonant passive components and can operate at a bandwidth exceeding 14 GHz, limited only by the speed of the modulators and photodetectors. Given recent advances in hybrid integration of soliton microcombs at microwave line rates, ultra-low loss silicon nitride waveguides, and high speed on-chip detectors and modulators, our approach provides a path towards full CMOS wafer-scale integration of the photonic tensor core. While we focus on convolution processing, more generally our results indicate the major potential of integrated photonics for parallel, fast, and efficient computational hardware in demanding AI applications such as autonomous driving, live video processing, and next generation cloud computing services.

References in corpus (21)

• Deep Learning with Coherent Nanophotonic Circuits
• All-Optical Machine Learning Using Diffractive Deep Neural Networks
• Temporal solitons in optical microresonators
• All-optical signal processing platforms for CMOS compatible integrated nonlinear optics
• Microresonator solitons for massively parallel coherent optical communications
• An Integrated-Photonics Optical-Frequency Synthesizer
• Neuromorphic computing with multi-memristive synapses
• O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis
• Fully integrated ultra-low power Kerr comb generation
• Massively parallel coherent laser ranging using soliton microcombs
• Attojoule Optoelectronics for Low-Energy Information Processing and Communications: a Tutorial Review
• Universal dynamics and controlled switching of dissipative Kerr solitons in optical microresonators
• Mixed-Precision In-Memory Computing
• Accurate deep neural network inference using computational phase-change memory
• Photonic Damascene Process for Integrated High-Q Microresonator Based Nonlinear Photonics
• A silicon photonic modulator neuron
• Dynamics of soliton crystals in optical microresonators
• Digital Electronics and Analog Photonics for Convolutional Neural Networks (DEAP-CNNs)
• Ultralow-power chip-based soliton microcombs for photonic integration
• An Optical Frontend for a Convolutional Neural Network
• Miniature Multi-Level Optical Memristive Switch Using Phase Change Material

Cited by in corpus (164)

• Single-chip photonic deep neural network for instantaneous image classification
• High performance photonic microwave filters based on a 50GHz optical soliton crystal Kerr micro-comb
• Bridging microcombs and silicon photonic engines for optoelectronics systems
• High-yield wafer-scale fabrication of ultralow-loss, dispersion-engineered silicon nitride photonic circuits
• The physics of optical computing
• High bandwidth temporal RF photonic signal processing with Kerr micro-combs: integration, fractional differentiation and Hilbert transforms
• Applications of integrated optical microcombs
• Intelligent Computing: The Latest Advances, Challenges and Future
• An optical neural network using less than 1 photon per multiplication
• Highly tunable broadband RF photonic fractional Hilbert transformer based on a Kerr soliton crystal microcomb source
• High-efficiency and broadband electro-optic frequency combs enabled by coupled micro-resonators
• Integral order photonic RF and microwave signal processors based on soliton crystal Kerr micro-combs
• Single chip photonic deep neural network with accelerated training
• Deep Learning with Coherent VCSEL Neural Networks
• Neuromorphic computing using wavelength-division multiplexing
• Quantum optics of soliton microcombs
• Scalable Optical Learning Operator
• Prospects and applications of photonic neural networks
• Review: Graphene oxide 2D thin films for high performance nonlinear integrated photonics
• High-order tensor flow processing using integrated photonic circuits
• Sagnac interference in integrated photonics for reflection mirrors, gyroscopes, filters, and wavelength interleavers
• In-memory photonic dot-product engine with electrically programmable weight banks
• Frequency comb distillation for optical superchannel transmission
• Non-volatile electrically programmable integrated photonics with a 5-bit operation
• Platicon microcomb generation using laser self-injection locking
• A 12-mode Universal Photonic Processor for Quantum Information Processing
• Optical coherent dot-product chip for sophisticated deep learning regression
• A power-efficient integrated lithium niobate electro-optic comb generator
• Polarization Multiplexed Diffractive Computing: All-Optical Implementation of a Group of Linear Transformations Through a Polarization-Encoded Diffractive Network
• Photonics for Neuromorphic Computing: Fundamentals, Devices, and Opportunities
• Theory for the Accuracy of Microcomb Photonic Microwave Transversal Signal Processors
• On-demand harnessing of photonic soliton molecules
• Broadband nonvolatile electrically programmable silicon photonic switches
• Scalable massively parallel computing using continuous-time data representation in nanoscale crossbar array
• Photonic extreme learning machine by free-space optical propagation
• Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment
• Inverse-designed low-index-contrast structures on silicon photonics platform for vector-matrix multiplication
• Massively Parallel Universal Linear Transformations using a Wavelength-Multiplexed Diffractive Optical Network
• 20-Mode Universal Quantum Photonic Processor
• Nonlinear co-generation of graphene plasmons for optoelectronic logic operations
• Arbitrary linear transformations for photons in the frequency synthetic dimension
• Optical Memory, Switching, and Neuromorphic Functionality in Metal Halide Perovskite Materials and Devices
• Synthetic-reflection self-injection-locked microcombs
• Direct (3+1)D laser writing of graded-index optical elements
• Integrated Triply Resonant Electro-Optic Frequency Comb in Lithium Tantalate
• Broadband quantum-dot frequency-modulated comb laser
• Asymptotically Fault-Tolerant Programmable Photonics
• Integrated Photonic Tensor Processing Unit for a Matrix Multiply: a Review
• An Electro-Photonic System for Accelerating Deep Neural Networks
• Octave-spanning Kerr soliton frequency combs in dispersion- and dissipation-engineered lithium niobate microresonators
• High-speed tunable microwave-rate soliton microcomb
• Data class-specific all-optical transformations and encryption
• Dispersion-less Kerr solitons in spectrally confined optical cavities
• A comprehensive review on developments of synthetic dimensions
• Understanding and mitigating noise in trained deep neural networks
• Coherent photonic crossbar as a universal linear operator
• Optical Generative Models
• Deep Photonic Reservoir Computer Based on Frequency Multiplexing with Fully Analog Connection Between Layers
• Thermal control of Kerr microresonator soliton comb via an optical sideband
• Scaling optical computing in synthetic frequency dimension using integrated cavity acousto-optics
• Sideband Injection Locking in Microresonator Frequency Combs
• Structural phase transitions between layered Indium Selenide for inte-grated photonic memory
• Scalability of all-optical neural networks based on spatial light modulators
• Photonic Extreme Learning Machine based on frequency multiplexing
• Foundry compatible, efficient wafer-scale manufacturing of ultra-low loss, high-density Si${}_3$N${}_4$ photonic integrated circuits
• Nonlocal phase-change metaoptics for reconfigurable nonvolatile image processing
• Radio-Frequency Multiply-And-Accumulate Operations with Spintronic Synapses
• Photonic reservoir computer based on frequency multiplexing
• Neural Schrödinger Equation:Physical Law as Neural Network
• Free-electron interaction with nonlinear optical states in microresonators
• Pyramid diffractive optical networks for unidirectional image magnification and demagnification
• Knowledge Distillation Circumvents Nonlinearity for Optical Convolutional Neural Networks
• Monadic Pavlovian associative learning in a backpropagation-free photonic network
• Free-form micro-optics enabling ultra-broadband low-loss fiber-to-chip coupling
• Energy-efficient neural network inference with microcavity exciton-polaritons
• A photonic complex perceptron for ultrafast data processing
• Multiscale simulations of growth-dominated Sb${}_2$Te phase-change material for non-volatile photonic applications
• Training neural networks with end-to-end optical backpropagation
• Programmable wave-based analog computing machine: a metastructure that designs metastructures
• Photonic topological phase transition induced by material phase transition
• Quantum-limited stochastic optical neural networks operating at a few quanta per activation
• Dissipative structures in topological lattices of nonlinear optical resonators
• Ultrafast single-channel machine vision based on neuro-inspired photonic computing
• Coherent terahertz radiation with 2.8-octave tunability through chip-scale photomixed microresonator optical parametric oscillation
• Information processing at the speed of light
• Photonic reservoir computing enabled by stimulated Brillouin scattering
• Robust and Efficient Single-Pixel Image Classificationwith Nonlinear Optics
• Vibrational Kerr solitons in an optomechanical microresonator
• Programmable access to microresonator solitons with modulational sideband heating
• Bonding nature and optical contrast of $TiT{e}_2$/$S{b}_{2}T{e}_3$ phase-change heterostructure
• Deep Photonic Networks with Arbitrary and Broadband Functionality
• Mechanically actuated Kerr soliton microcombs
• All-optical nonlinear activation function based on stimulated Brillouin scattering
• Symmetric silicon microring resonator optical crossbar array for accelerated inference and training in deep learning
• An optoacoustic field-programmable perceptron for recurrent neural networks
• Optical microcombs in whispering gallery mode crystalline resonators with dispersive intermode interactions
• The performance of microwave photonic signal processors based on microcombs with different input signal waveforms
• Microwave photonic transversal filters based on microcombs with feedback control
• Ultra-high endurance silicon photonic memory using vanadium dioxide
• Programmable metasurfaces for future photonic artificial intelligence
• Soliton Microcomb Generation in a III-V Photonic Crystal Cavity
• Contrasting the relative performance of RF photonic transversal signal processors based on microcombs using discrete components versus integrated devices
• Training Coupled Phase Oscillators as a Neuromorphic Platform using Equilibrium Propagation
• Optical Neural Network Based on Synthetic Nonlinear Photonic Lattices
• Waveguide-multiplexed photonic matrix-vector multiplication processor using multiport photodetectors
• Soliton microcombs in X-cut LiNbO3 microresonators
• Emergent self-adaptation in an integrated photonic neural network for backpropagation-free learning
• Deep Photonic Reservoir Computer for Speech Recognition
• Modulation instability of Kerr optical frequency combs in dual-coupled optical cavities
• Asymmetrical estimator for training encapsulated deep photonic neural networks
• Noise mitigation strategies in physical feedforward neural networks
• Arrayed waveguide gratings in lithium tantalate integrated photonics
• 16-channel Photonic Solver for Optimization Problems on a Silicon Chip
• Integrated Photonic Fractional Convolution Accelerator
• Copper-impurity-free photonic integrated circuits enable deterministic soliton microcombs
• Computing with injection-locked spintronic diodes
• Integrated and DC-powered superconducting microcomb
• Photonic kernel machine learning for ultrafast spectral analysis
• AnalogVNN: A fully modular framework for modeling and optimizing photonic neural networks
• A Blueprint for Precise and Fault-Tolerant Analog Neural Networks
• Lasing on hybridized soliton frequency combs
• Braided interferometer mesh for robust photonic matrix-vector multiplications with non-ideal components
• Hardware-Efficient Photonic Tensor Core: Accelerating Deep Neural Networks with Structured Compression
• Special Session: Neuromorphic hardware design and reliability from traditional CMOS to emerging technologies
• Neural Network Methods for Radiation Detectors and Imaging
• Single-Shot Matrix-Matrix Multiplication Optical Tensor Processor for Deep Learning
• High-precision programming of large-scale ring resonator circuits with minimal pre-calibration
• Optical neural network architecture for deep learning with the temporal synthetic dimension
• Arbitrary control over multimode wave propagation for machine learning
• A high-fidelity and large-scale reconfigurable photonic processor for NISQ applications
• Conflict-free joint sampling for preference satisfaction through quantum interference
• Solving multi-armed bandit problems using a chaotic microresonator comb
• Temporal dissipative structures in optical Kerr resonators with transient loss fluctuation
• A 262 TOPS Hyperdimensional Photonic AI Accelerator powered by a Si3N4 microcomb laser
• Photonic Neuromorphic Accelerators for Event-Based Imaging Flow Cytometry
• An All-Optical General-Purpose CPU and Optical Computer Architecture
• Design of an ultra-compact, energy-efficient non-volatile photonic switch based on phase change materials
• Contemporary implementations of spiking bio-inspired neural networks
• Conflict-free joint decision by lag and zero-lag synchronization in laser network
• SUANPAN: Scalable Photonic Linear Vector Machine
• Blending Optimal Control and Biologically Plausible Learning for Noise-Robust Physical Neural Networks
• Scalable intensity-based photonic matrix-vector multiplication processor using single-wavelength time-division-multiplexed signals
• Photonic systolic array for all-optical matrix-matrix multiplication
• Photonic single perceptron at Giga-OP/s speeds with Kerr microcombs for scalable optical neural networks
• Systematic dispersion engineering of crystalline microresonators for broadband and coherent frequency comb generation
• Passive All-Optical Nonlinear Neuron Activation via PPLN Nanophotonic Waveguides
• Self-cooling, blue-detuned dissipative Kerr microresonator soliton comb
• Partial Differential Equations is All You Need for Generating Neural Architectures -- A Theory for Physical Artificial Intelligence Systems
• Quantum-secure multiparty deep learning
• Harnessing Optoelectronic Noises in a Photonic Generative Network
• Online training and pruning of multi-wavelength photonic neural networks
• Gradient descent in materia through homodyne gradient extraction
• Metalens array for complex-valued optical discrete Fourier transform
• Internal noise in hardware deep and recurrent neural networks helps with learning
• Single-mode emission by phase-delayed coupling between nano-lasers
• Sub-quadratic scalable approximate linear converter using multi-plane light conversion with low-entropy mode mixers
• Mapping ultrafast timing jitter in dispersion-managed 89 GHz frequency microcombs via self-heterodyne linear interferometry
• Slimmed optical neural networks with multiplexed neuron sets and a corresponding backpropagation training algorithm
• Perspective of high-speed Mach-Zehnder modulators based on nonlinear optics and complex band structures
• Resonant tunnelling diode nano-optoelectronic spiking nodes for neuromorphic information processing
• 2022 Roadmap on Neuromorphic Computing and Engineering
• All-Optical Doubly Resonant Cavities for ReLU Function in Nanophotonic Deep Learning
• Polaritonic Machine Learning for Graph-based Data Analysis
• Centi-combs: Low-noise sub-GHz repetition-rate soliton frequency combs from crystalline resonators