DeepSTORM3D: dense three dimensional localization microscopy and point spread function design by deep learning

Abstract

Localization microscopy is an imaging technique in which the positions of individual nanoscale point emitters (e.g. fluorescent molecules) are determined at high precision from their images. This is the key ingredient in single/multiple-particle-tracking and several super-resolution microscopy approaches. Localization in three-dimensions (3D) can be performed by modifying the image that a point-source creates on the camera, namely, the point-spread function (PSF). The PSF is engineered using additional optical elements to vary distinctively with the depth of the point-source. However, localizing multiple adjacent emitters in 3D poses a significant algorithmic challenge, due to the lateral overlap of their PSFs. Here, we train a neural network to receive an image containing densely overlapping PSFs of multiple emitters over a large axial range and output a list of their 3D positions. Furthermore, we then use the network to design the optimal PSF for the multi-emitter case. We demonstrate our approach numerically as well as experimentally by 3D STORM imaging of mitochondria, and volumetric imaging of dozens of fluorescently-labeled telomeres occupying a mammalian nucleus in a single snapshot.

main text: 9 pages, 5 figures, supplementary information: 29 pages, 20 figures

References in corpus (10)

• Phase recovery and holographic image reconstruction using deep learning in neural networks
• Deep-STORM: super-resolution single-molecule microscopy by deep learning
• PhaseStain: Digital staining of label-free quantitative phase microscopy images using deep learning
• Deep learning approach to Fourier ptychographic microscopy
• Convolutional neural networks automate detection for tracking of submicron scale particles in 2D and 3D
• Learning Depth from Single Images with Deep Neural Network Embedding Focal Length
• Deep learning-based super-resolution in coherent imaging systems
• Light scattering control in transmission and reflection with neural networks
• cellSTORM - Cost-effective Super-Resolution on a Cellphone using dSTORM
• Fast 3D cell tracking with wide-field fluorescence microscopy through deep learning

Cited by in corpus (28)

• Deep learning in nano-photonics: inverse design and beyond
• Self-supervised learning of hologram reconstruction using physics consistency
• Intelligent Meta-Imagers: From Compressed to Learned Sensing
• Deep learning in biomedical optics
• Unidirectional Imaging using Deep Learning-Designed Materials
• Fundamental bounds on the precision of iSCAT, COBRI and dark-field microscopy for 3D localization and mass photometry
• Single-molecule orientation localization microscopy II: a performance comparison
• 100,000 frames-per-second compressive imaging with a conventional rolling-shutter camera by random point-spread-function engineering
• Few-shot Transfer Learning for Holographic Image Reconstruction using a Recurrent Neural Network
• Multiplexed PSF engineering for 3D multicolor particle tracking
• Diffractive optical system design by cascaded propagation
• Deep learning-based virtual refocusing of images using an engineered point-spread function
• Recognizing three-dimensional phase images with deep learning
• Neural Network-Based Processing and Reconstruction of Compromised Biophotonic Image Data
• Deep Optics for Video Snapshot Compressive Imaging
• Integration of Programmable Diffraction with Digital Neural Networks
• XLuminA: An Auto-differentiating Discovery Framework for Super-Resolution Microscopy
• Towards optimal point spread function design for resolving closely spaced emitters in three dimensions
• Universal point spread function engineering for 3D optical information processing
• Designing lensless imaging systems to maximize information capture
• Point-Spread-Function Engineering in MINFLUX: Optimality of Donut and Half-Moon Excitation Patterns
• Dipole-Spread Function Engineering for 6D Super-Resolution Microscopy
• Collaborative On-Sensor Array Cameras
• Diffractive Interconnects: All-Optical Permutation Operation Using Diffractive Networks
• Advancing biological super-resolution microscopy through deep learning: a brief review
• Universal computational super-resolution framework for imaging of quantum dots
• Implicitly Learned Neural Phase Functions for Basis-Free Point Spread Function Engineering
• PiLocNet: Physics-informed neural network on 3D localization with rotating point spread function