Image sensing with multilayer, nonlinear optical neural networks

Abstract

Optical imaging is commonly used for both scientific and technological applications across industry and academia. In image sensing, a measurement, such as of an object's position, is performed by computational analysis of a digitized image. An emerging image-sensing paradigm breaks this delineation between data collection and analysis by designing optical components to perform not imaging, but encoding. By optically encoding images into a compressed, low-dimensional latent space suitable for efficient post-analysis, these image sensors can operate with fewer pixels and fewer photons, allowing higher-throughput, lower-latency operation. Optical neural networks (ONNs) offer a platform for processing data in the analog, optical domain. ONN-based sensors have however been limited to linear processing, but nonlinearity is a prerequisite for depth, and multilayer NNs significantly outperform shallow NNs on many tasks. Here, we realize a multilayer ONN pre-processor for image sensing, using a commercial image intensifier as a parallel optoelectronic, optical-to-optical nonlinear activation function. We demonstrate that the nonlinear ONN pre-processor can achieve compression ratios of up to 800:1 while still enabling high accuracy across several representative computer-vision tasks, including machine-vision benchmarks, flow-cytometry image classification, and identification of objects in real scenes. In all cases we find that the ONN's nonlinearity and depth allowed it to outperform a purely linear ONN encoder. Although our experiments are specialized to ONN sensors for incoherent-light images, alternative ONN platforms should facilitate a range of ONN sensors. These ONN sensors may surpass conventional sensors by pre-processing optical information in spatial, temporal, and/or spectral dimensions, potentially with coherent and quantum qualities, all natively in the optical domain.

References in corpus (4)

• Photonics for artificial intelligence and neuromorphic computing
• 11 TeraFLOPs per second photonic convolutional accelerator for deep learning optical neural networks
• An Optical Frontend for a Convolutional Neural Network
• Neural networks based on ultrafast time-delayed effects in exciton-polaritons

Cited by in corpus (16)

• The physics of optical computing
• Photonics for Neuromorphic Computing: Fundamentals, Devices, and Opportunities
• Nonlinear optical encoding enabled by recurrent linear scattering
• All-optical image denoising using a diffractive visual processor
• Optical Generative Models
• Ultrafast single-channel machine vision based on neuro-inspired photonic computing
• Measurement of Near-field Thermal Radiation between Multilayered Metamaterials
• Programmable metasurfaces for future photonic artificial intelligence
• Efficient option pricing with unary-based photonic computing chip and generative adversarial learning
• Transferable polychromatic optical encoder for neural networks
• Neural Network Methods for Radiation Detectors and Imaging
• High-performance real-world optical computing trained by in situ gradient-based model-free optimization
• Broadband Unidirectional Visible Imaging Using Wafer-Scale Nano-Fabrication of Multi-Layer Diffractive Optical Processors
• Compute-first optical detection for noise-resilient visual perception
• Quantum-secure multiparty deep learning
• Polaritonic Machine Learning for Graph-based Data Analysis