Synthetic Depth-of-Field with a Single-Camera Mobile Phone
arXiv:1806.04171 · doi:10.1145/3197517.3201329
Abstract
Shallow depth-of-field is commonly used by photographers to isolate a subject from a distracting background. However, standard cell phone cameras cannot produce such images optically, as their short focal lengths and small apertures capture nearly all-in-focus images. We present a system to computationally synthesize shallow depth-of-field images with a single mobile camera and a single button press. If the image is of a person, we use a person segmentation network to separate the person and their accessories from the background. If available, we also use dense dual-pixel auto-focus hardware, effectively a 2-sample light field with an approximately 1 millimeter baseline, to compute a dense depth map. These two signals are combined and used to render a defocused image. Our system can process a 5.4 megapixel image in 4 seconds on a mobile phone, is fully automatic, and is robust enough to be used by non-experts. The modular nature of our system allows it to degrade naturally in the absence of a dual-pixel sensor or a human subject.
arXiv commentAccepted to SIGGRAPH 2018. Basis for Portrait Mode on Google Pixel 2 and Pixel 2 XL
References in corpus (5)
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
- Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
- Pose2Instance: Harnessing Keypoints for Person Instance Segmentation
Cited by in corpus (35)
- Single Image Portrait Relighting
- Handheld Multi-Frame Super-Resolution
- Whole-Slide Image Focus Quality: Automatic Assessment and Impact on AI Cancer Detection
- Computational Imaging and Artificial Intelligence: The Next Revolution of Mobile Vision
- Portrait Neural Radiance Fields from a Single Image
- DoF-NeRF: Depth-of-Field Meets Neural Radiance Fields
- Depth-aware Blending of Smoothed Images for Bokeh Effect Generation
- BGGAN: Bokeh-Glass Generative Adversarial Network for Rendering Realistic Bokeh
- Using Scene Graph Context to Improve Image Generation
- NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images
- Defocus to focus: Photo-realistic bokeh rendering by fusing defocus and radiance priors
- Correcting Face Distortion in Wide-Angle Videos
- Consistent Video Depth Estimation
- FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras
- AIM 2020 Challenge on Rendering Realistic Bokeh
- KOALAnet: Blind Super-Resolution using Kernel-Oriented Adaptive Local Adjustment
- Synthetic Defocus and Look-Ahead Autofocus for Casual Videography
- Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration
- Dual-Camera All-in-Focus Neural Radiance Fields
- Natural & Adversarial Bokeh Rendering via Circle-of-Confusion Predictive Network
- Scale-Invariant Monocular Depth Estimation via SSI Depth
- Sky Optimization: Semantically aware image processing of skies in low-light photography
- Apple's Synthetic Defocus Noise Pattern: Characterization and Forensic Applications
- DuNet: Learning Depth Estimation from Dual-Cameras and Dual-Pixels
- BaMBNet: A Blur-aware Multi-branch Network for Defocus Deblurring
- Neural Imaging Pipelines - the Scourge or Hope of Forensics?
- Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image
- Bokeh Diffusion: Defocus Blur Control in Text-to-Image Diffusion Models
- Learned Dual-View Reflection Removal
- Computational Long Exposure Mobile Photography
- Facial Depth and Normal Estimation using Single Dual-Pixel Camera
- Unsupervised Learning of Depth and Depth-of-Field Effect from Natural Images with Aperture Rendering Generative Adversarial Networks
- Single image deep defocus estimation and its applications
- Neural Camera Simulators
- 3D Ken Burns Effect from a Single Image