All-Optical Machine Learning Using Diffractive Deep Neural Networks

Abstract

We introduce an all-optical Diffractive Deep Neural Network (D2NN) architecture that can learn to implement various functions after deep learning-based design of passive diffractive layers that work collectively. We experimentally demonstrated the success of this framework by creating 3D-printed D2NNs that learned to implement handwritten digit classification and the function of an imaging lens at terahertz spectrum. With the existing plethora of 3D-printing and other lithographic fabrication methods as well as spatial-light-modulators, this all-optical deep learning framework can perform, at the speed of light, various complex functions that computer-based neural networks can implement, and will find applications in all-optical image analysis, feature detection and object classification, also enabling new camera designs and optical components that can learn to perform unique tasks using D2NNs.

20 pages, 4 figures

References in corpus (8)

• A Survey on Deep Learning in Medical Image Analysis
• Deep Learning with Coherent Nanophotonic Circuits
• Deep Convolutional Neural Network for Inverse Problems in Imaging
• Phase recovery and holographic image reconstruction using deep learning in neural networks
• Deep Learning Microscopy
• Theory of graphene saturable absorption
• Deep learning enhanced mobile-phone microscopy
• Trainable and Dynamic Computing: Error Backpropagation through Physical Media

Cited by in corpus (296)

• Machine learning and the physical sciences
• Parallel convolution processing using an integrated photonic tensor core
• Photonics for artificial intelligence and neuromorphic computing
• 11 TeraFLOPs per second photonic convolutional accelerator for deep learning optical neural networks
• Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems
• Single-chip photonic deep neural network for instantaneous image classification
• Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit
• Deep learning in nano-photonics: inverse design and beyond
• All Optical Neural Network with Nonlinear Activation Functions
• All-Optically Controlled Memristor
• Large-Scale Optical Neural Networks based on Photoelectric Multiplication
• Wave Physics as an Analog Recurrent Neural Network
• High bandwidth temporal RF photonic signal processing with Kerr micro-combs: integration, fractional differentiation and Hilbert transforms
• Optical secret sharing with cascaded metasurface holography
• Intelligent Nanophotonics: Merging Photonics and Artificial Intelligence at the Nanoscale
• Applications of integrated optical microcombs
• Design of Task-Specific Optical Systems Using Broadband Diffractive Neural Networks
• Intelligent Computing: The Latest Advances, Challenges and Future
• An optical neural network using less than 1 photon per multiplication
• Analysis of Diffractive Optical Neural Networks and Their Integration with Electronic Neural Networks
• On the use of deep learning for phase recovery
• Machine-Learning-Assisted Metasurface Design for High-Efficiency Thermal Emitter Optimization
• Spectrally-Encoded Single-Pixel Machine Vision Using Diffractive Networks
• Deep Learning with Coherent VCSEL Neural Networks
• Neuromorphic computing using wavelength-division multiplexing
• Tailoring Structure-borne Sound Through Bandgap Engineering in Phononic Crystals and Metamaterials: A Comprehensive Review
• Theory of neuromorphic computing by waves: machine learning by rogue waves, dispersive shocks, and solitons
• Photonic tensor cores for machine learning
• Prospects and applications of photonic neural networks
• Terahertz Pulse Shaping Using Diffractive Surfaces
• Computational Imaging Without a Computer: Seeing Through Random Diffusers at the Speed of Light
• Class-specific Differential Detection in Diffractive Optical Neural Networks Improves Inference Accuracy
• Intelligent metaphotonics empowered by machine learning
• In-memory photonic dot-product engine with electrically programmable weight banks
• Meta-optics for spatial optical analog computing
• All-Optical Information Processing Capacity of Diffractive Surfaces
• Misalignment Resilient Diffractive Optical Networks
• Minireview on Disordered Optical Metasurfaces
• Ensemble learning of diffractive optical networks
• Controlling mass and energy diffusion with metamaterials
• Self-learning photonic signal processor with an optical neural network chip
• Heuristic Recurrent Algorithms for Photonic Ising Machines
• Optical machine learning with incoherent light and a single-pixel detector
• Roadmap on structured waves
• Deep Learning in Mobile and Wireless Networking: A Survey
• Polarization Multiplexed Diffractive Computing: All-Optical Implementation of a Group of Linear Transformations Through a Polarization-Encoded Diffractive Network
• Polaritonic neuromorphic computing outperforms linear classifiers
• Snapshot Multispectral Imaging Using a Diffractive Optical Network
• An Optical Frontend for a Convolutional Neural Network
• Meta-neural-network for Realtime and Passive Deep-learning-based Object Recognition
• Learning without neurons in physical systems
• Photonics for Neuromorphic Computing: Fundamentals, Devices, and Opportunities
• Why optics needs thickness
• Computational inverse design for cascaded systems of metasurface optics
• Scalable massively parallel computing using continuous-time data representation in nanoscale crossbar array
• Photonic extreme learning machine by free-space optical propagation
• Free-space optical neural network based on thermal atomic nonlinearity
• Nonlinear optical encoding enabled by recurrent linear scattering
• Diffractive all-optical computing for quantitative phase imaging
• Inverse-designed low-index-contrast structures on silicon photonics platform for vector-matrix multiplication
• Intelligent Meta-Imagers: From Compressed to Learned Sensing
• Massively Parallel Universal Linear Transformations using a Wavelength-Multiplexed Diffractive Optical Network
• To image, or not to image: Class-specific diffractive cameras with all-optical erasure of undesired objects
• Predicting the dynamic process and model parameters of the vector optical solitons in birefringent fibers via the modified PINN
• Roadmap for Unconventional Computing with Nanotechnology
• Nonlinear co-generation of graphene plasmons for optoelectronic logic operations
• Scale-, shift- and rotation-invariant diffractive optical networks
• Vector Symbolic Architectures as a Computing Framework for Emerging Hardware
• Arbitrary linear transformations for photons in the frequency synthetic dimension
• Applications and Techniques for Fast Machine Learning in Science
• Magnet-free nonreciprocal metasurface for on-demand bi-directional phase modulation
• Stacked Intelligent Metasurface Enabled LEO Satellite Communications Relying on Statistical CSI
• Rapid Sensing of Hidden Objects and Defects using a Single-Pixel Diffractive Terahertz Processor
• Efficient training and design of photonic neural network through neuroevolution
• All-optical image classification through unknown random diffusers using a single-pixel diffractive network
• Universal Linear Intensity Transformations Using Spatially-Incoherent Diffractive Processors
• Plasmonic photoconductive terahertz focal-plane array with pixel super-resolution
• Super-resolution image display using diffractive decoders
• Soliton crystal Kerr microcombs for high-speed, scalable optical neural networks at 10 GigaOPs/s
• All-Optical Spiking Neuron Based On Passive Micro-Resonator
• An Electro-Photonic System for Accelerating Deep Neural Networks
• Channel Estimation for Stacked Intelligent Metasurface-Assisted Wireless Networks
• All-optical image denoising using a diffractive visual processor
• Data class-specific all-optical transformations and encryption
• Multiple-image encryption and hiding with an optical diffractive neural network
• Multiplane Quantitative Phase Imaging Using a Wavelength-Multiplexed Diffractive Optical Processor
• Coupled nonlinear delay systems as deep convolutional neural networks
• Unidirectional Imaging using Deep Learning-Designed Materials
• Classification of time-domain waveforms using a speckle-based optical reservoir computer
• Stacked Intelligent Metasurfaces for Multiuser Downlink Beamforming in the Wave Domain
• Understanding and mitigating noise in trained deep neural networks
• Optical Generative Models
• Cascadable all-optical NAND gates using diffractive networks
• High-Speed Phase-Only Spatial Light Modulators with Two-Dimensional Tunable Microcavity Arrays
• All-optical complex field imaging using diffractive processors
• Deep Photonic Reservoir Computer Based on Frequency Multiplexing with Fully Analog Connection Between Layers
• Diffractive deep neural network based adaptive optics scheme for vortex beam in oceanic turbulence
• Transmit Beamforming Design for ISAC with Stacked Intelligent Metasurfaces
• Polarized deep diffractive neural network for classification, generation, multiplexing and de-multiplexing of orbital angular momentum modes
• Photonic convolutional neural networks using integrated diffractive optics
• Scaling optical computing in synthetic frequency dimension using integrated cavity acousto-optics
• An extended physics informed neural network for preliminary analysis of parametric optimal control problems
• Developing of a photonic hardware platform for brain-inspired computing based on $5\times 5$ VCSEL arrays
• Scalability of all-optical neural networks based on spatial light modulators
• Universal Polarization Transformations: Spatial programming of polarization scattering matrices using a deep learning-designed diffractive polarization transformer
• Far-field subwavelength acoustic imaging by deep learning
• Classification and reconstruction of spatially overlapping phase images using diffractive optical networks
• Optical computation of a spin glass dynamics with tunable complexity
• Information hiding cameras: optical concealment of object information into ordinary images
• Photonic Extreme Learning Machine based on frequency multiplexing
• Quadrature Photonic Spatial Ising Machine
• Small-brain neural networks rapidly solve inverse problems with vortex Fourier encoders
• Learning Diffractive Optical Communication Around Arbitrary Opaque Occlusions
• Quantitative phase imaging (QPI) through random diffusers using a diffractive optical network
• Simultaneously sorting vector vortex beams of 120 modes
• Fundamental aspects of noise in analog-hardware neural networks
• How to build the optical inverse of a multimode fibre
• Neural network design of multilayer metamaterial for temporal differentiation
• Ultrahigh-fidelity spatial mode quantum gates in high-dimensional space by diffractive deep neural networks
• Photonic reservoir computer based on frequency multiplexing
• Resource-efficient photonic quantum computation with high-dimensional cluster states
• Neural Schrödinger Equation:Physical Law as Neural Network
• Subwavelength Imaging using a Solid-Immersion Diffractive Optical Processor
• Multispectral Quantitative Phase Imaging Using a Diffractive Optical Network
• Pyramid diffractive optical networks for unidirectional image magnification and demagnification
• Diffractive coupling for photonic networks: how big can we go?
• Knowledge Distillation Circumvents Nonlinearity for Optical Convolutional Neural Networks
• Spatial Mode Correction of Single Photons using Machine Learning
• Physical deep learning based on optimal control of dynamical systems
• Simultaneously sorting overlapping quantum states of light
• Energy-efficient neural network inference with microcavity exciton-polaritons
• Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation
• A Broad-Spectrum Diffractive Network via Ensemble Learning
• Wave-based extreme deep learning based on non-linear time-Floquet entanglement
• Training neural networks with end-to-end optical backpropagation
• Programmable wave-based analog computing machine: a metastructure that designs metastructures
• Complex-valued universal linear transformations and image encryption using spatially incoherent diffractive networks
• Quantum-limited stochastic optical neural networks operating at a few quanta per activation
• On-demand Photonic Ising Machine with Simplified Hamiltonian Calculation by Phase encoding and Intensity Detection
• Inverse Design of Multi-input Multi-output 2D Metastructured Devices
• Broadband nonlinear modulation of incoherent light using a transparent optoelectronic neuron array
• Ultrafast single-channel machine vision based on neuro-inspired photonic computing
• Tackling Sampling Noise in Physical Systems for Machine Learning Applications: Fundamental Limits and Eigentasks
• Supervised learning of an opto-magnetic neural network with ultrashort laser pulses
• Silicon photonic-electronic neural network for fibre nonlinearity compensation
• All-Optical Phase Conjugation Using Diffractive Wavefront Processing
• Time-lapse image classification using a diffractive neural network
• Diffractive optical system design by cascaded propagation
• Complete vectorial optical mode converter using multi-layer metasurface
• Deep Photonic Networks with Arbitrary and Broadband Functionality
• Hyperparameter tuning of optical neural network classifiers for high-order gaussian beams
• Experimental reservoir computing with diffractively coupled VCSELs
• Design of stacked intelligent metasurfaces with reconfigurable amplitude and phase for multiuser downlink beamforming
• Optical information transfer through random unknown diffusers using electronic encoding and diffractive decoding
• Referenceless characterisation of complex media using physics-informed neural networks
• Single-Pixel Pattern Recognition with Coherent Nonlinear Optics
• Phase Retrieval and Design with Automatic Differentiation
• Multiplexed all-optical permutation operations using a reconfigurable diffractive optical network
• An optoacoustic field-programmable perceptron for recurrent neural networks
• Programmable metasurfaces for future photonic artificial intelligence
• Analysis of Diffractive Neural Networks for Seeing Through Random Diffusers
• Mechanical intelligence via fully reconfigurable elastic neuromorphic metasurfaces
• Stacked Intelligent Metasurface Enabled Near-Field Multiuser Beamfocusing in the Wave Domain
• Deep Neural Network Inverse Design of Integrated Nanophotonic Devices
• Transferable polychromatic optical encoder for neural networks
• All-Optical Image Identification with Programmable Matrix Transformation
• ProdSumNet: reducing model parameters in deep neural networks via product-of-sums matrix decompositions
• Waveguide-multiplexed photonic matrix-vector multiplication processor using multiport photodetectors
• Spatial-photonic Ising machine by space-division multiplexing with physically tunable coefficients of a multi-component model
• Optical Phase Dropout in Diffractive Deep Neural Network
• Noise mitigation strategies in physical feedforward neural networks
• Boolean learning under noise-perturbations in hardware neural networks
• Asymmetrical estimator for training encapsulated deep photonic neural networks
• Integration of Programmable Diffraction with Digital Neural Networks
• Optical convolutional neural network with atomic nonlinearity
• 16-channel Photonic Solver for Optimization Problems on a Silicon Chip
• A Novel Modeling Approach for All-Dielectric Metasurfaces Using Deep Neural Networks
• Decision-making and control with diffractive optical networks
• Photonic kernel machine learning for ultrafast spectral analysis
• Inverse design of ultracompact multi-focal optical devices by diffractive neural networks
• Intelligent mechanical metamaterials towards learning static and dynamic behaviors
• AnalogVNN: A fully modular framework for modeling and optimizing photonic neural networks
• Design of ultracompact broadband focusing spectrometers based on deep diffractive neural networks
• Axial correlation revivals and number factorization with structured random waves
• Universal classical optical computing inspired by quantum information process
• Optoelectronic generative adversarial networks
• Predicting mode-locked fiber laser output using feed-forward neural network
• Deep optical neural network by living tumour brain cells
• High-performance real-world optical computing trained by in situ gradient-based model-free optimization
• Neural Network Methods for Radiation Detectors and Imaging
• Joint SIM Configuration and Power Allocation for Stacked Intelligent Metasurface-assisted MU-MISO Systems with TD3
• Impact of white noise in artificial neural networks trained for classification: performance and noise mitigation strategies
• Universal point spread function engineering for 3D optical information processing
• Express Wavenet -- a low parameter optical neural network with random shift wavelet pattern
• Image classification using collective optical modes of an array of nanolasers
• Multiplexed Metasurfaces for Diffractive Optics via Phase Correlation Method
• Neural networks based on ultrafast time-delayed effects in exciton-polaritons
• Polarization and Orbital Angular Momentum Encoded Quantum Toffoli Gate Enabled by Diffractive Neural Networks
• Lossless, Complex-Valued Optical Field Control with Compound Metaoptics
• Backpropagation through nonlinear units for all-optical training of neural networks
• High-speed coherent photonic random-access memory in long-lasting sound waves
• Optical Neural Ordinary Differential Equations
• A Reconfigurable Linear RF Analog Processor for Realizing Microwave Artificial Neural Network
• Meta-operators for all-optical image processing
• Massively parallel and universal approximation of nonlinear functions using diffractive processors
• Rapid characterisation of linear-optical networks via PhaseLift
• Optical neural network architecture for deep learning with the temporal synthetic dimension
• An optical diffractive deep neural network with multiple frequency-channels
• Minimalist Vision with Freeform Pixels
• Conflict-free joint sampling for preference satisfaction through quantum interference
• Scaling Achievable Rates in SIM-aided MIMO Systems with Metasurface Layers: A Hybrid Optimization Framework
• Integrated Photonic Programmable Random Matrix Generator with Minimal Active Components
• Unidirectional imaging with partially coherent light
• Broadband Unidirectional Visible Imaging Using Wafer-Scale Nano-Fabrication of Multi-Layer Diffractive Optical Processors
• A Low Power Threshold, Ultrathin Optical Limiter Based on a Nonlinear Zone Plate
• Physical Computing for Materials Acceleration Platforms
• Light structuring via nonlinear total angular momentum addition with flat optics
• Planar Fourier Optics for Slab Waveguides, Surface Plasmon Polaritons and 2D Materials
• Optimizing Structured Surfaces for Diffractive Waveguides
• SUANPAN: Scalable Photonic Linear Vector Machine
• Heterogeneous networks for phase-sensitive engineering of optical disordered materials
• Model-free Optical Processors using In Situ Reinforcement Learning with Proximal Policy Optimization
• Training and pattern recognition by an opto-magnetic neural network
• Structural Vibration Monitoring with Diffractive Optical Processors
• Theory of Quantum Path Computing with Fourier Optics and Future Applications for Quantum Supremacy, Neural Networks and Nonlinear Schrödinger Equations
• Electro-optical Neural Networks based on Time-stretch Method
• Image amplification in a self-imaging degenerate optical cavity
• Photonic single perceptron at Giga-OP/s speeds with Kerr microcombs for scalable optical neural networks
• Photonic systolic array for all-optical matrix-matrix multiplication
• Coherence Awareness in Diffractive Neural Networks
• Impact of internal noise on convolutional neural networks
• Quantum optical classifier with superexponential speedup
• Free-Space Optical Spiking Neural Network
• Photonic neural field on a silicon chip: large-scale, high-speed neuro-inspired computing and sensing
• Nonlinear unitary circuits for photonic neural networks
• Simultaneous Sorting of Wavelengths and Spatial Modes using Multi-Plane Light Conversion
• Spin Hamiltonians in the Modulated Momenta of Light
• Analog-to-digital conversion revolutionized by deep learning
• Analog dual classifier via a time-modulated neuromorphic metasurface
• Information contraction in noisy binary neural networks and its implications
• Two-Stage Coded-Sliding Beam Training and QoS-Constrained Sum-Rate Maximization for SIM-Assisted Wireless Communications
• Self-configuring high-speed multi-plane light conversion
• AI based design of 2D material integrated optical polarizers
• Unidirectional focusing of light using structured diffractive surfaces
• Holographic photonic neuron
• Online training and pruning of multi-wavelength photonic neural networks
• Compute-first optical detection for noise-resilient visual perception
• Spatial Optical Simulator for Classical Statistical Models
• Unitary Learning for Deep Diffractive Neural Network
• Harnessing Optoelectronic Noises in a Photonic Generative Network
• Quantum-secure multiparty deep learning
• Performance evaluation of an integrated photonic convolutional neural network based on delay buffering and wavelength division multiplexing
• Integrated Photonic FFT for Optical Convolutions towards Efficient and High-Speed Neural Networks
• Metalens array for complex-valued optical discrete Fourier transform
• Computer-free, all-optical reconstruction of holograms using diffractive networks
• Noise impact on recurrent neural network with linear activation function
• Slimmed optical neural networks with multiplexed neuron sets and a corresponding backpropagation training algorithm
• Diffractive Interconnects: All-Optical Permutation Operation Using Diffractive Networks
• Building and aligning a 10-plane light converter
• BlurryScope enables compact, cost-effective scanning microscopy for HER2 scoring using deep learning on blurry images
• Neural networks with quantum states of light
• End-to-End Framework for Efficient Deep Learning Using Metasurfaces Optics
• Real-time image processing with a 2D semiconductor neural network vision sensor
• A single metagrating metastructure for wave-based parallel analog computing
• Optical spin glasses: a new model for glassy systems
• A Diffractive Neural Network with Weight-Noise-Injection Training
• High-Freedom Inverse Design with Deep Neural Network for Metasurface Filter in the Visible
• Interpretable inverse design of particle spectral emissivity using machine learning
• Single photonic perceptron based on a soliton crystal Kerr microcomb for high-speed, scalable, optical neural networks
• AI-Driven Design of Stacked Intelligent Metasurfaces for Software-Defined Radio Applications
• 2022 Roadmap on Neuromorphic Computing and Engineering
• All-Optical Synthesis of an Arbitrary Linear Transformation Using Diffractive Surfaces
• Phase-multiplexed optical computing: Reconfiguring a multi-task diffractive optical processor using illumination phase diversity
• A neuromorphic camera for tracking passive and active matter with lower data throughput
• A Comprehensive Survey on Nanophotonics Neural Networks
• Fast Autofocusing using Tiny Transformer Networks for Digital Holographic Microscopy
• 2D beam shaping via 1D spatial light modulation
• Implementation of transformer-based LLMs with large-scale optoelectronic neurons on a CMOS compatible platform
• Programming multi-level quantum gates in disordered computing reservoirs via machine learning and TensorFlow
• Modeling and benchmarking quantum optical neurons for efficient neural computation
• Numerical study of laser micro- and nano-processing of nanocomposite porous materials
• Broadband vectorial ultra-flat optics with experimental efficiency up to 99% in the visible via universal approximators
• On Learning from Ghost Imaging without Imaging
• Spectral splitting and concentration of broadband light using neural networks
• To Root Artificial Intelligence Deeply in Basic Science for a New Generation of AI
• Polaritonic Machine Learning for Graph-based Data Analysis
• High speed and reconfigurable optronic neural network with digital nonlinear activation
• All-Optical Doubly Resonant Cavities for ReLU Function in Nanophotonic Deep Learning
• Resonant tunnelling diode nano-optoelectronic spiking nodes for neuromorphic information processing
• Privacy-Preserving Image Acquisition Using Trainable Optical Kernel
• Random Graph-Based Neuromorphic Learning with a Layer-Weaken Structure
• Joint Image and Depth Estimation with Mask-Based Lensless Cameras
• Compatible Learning for Deep Photonic Neural Network
• Improvement of image classification by multiple optical scattering
• Interleaved diffractive networks for information transfer through random diffusers
• What Happens on the Edge, Stays on the Edge: Toward Compressive Deep Learning