Holistically-Nested Edge Detection

Abstract

We develop a new edge detection algorithm that tackles two important issues in this long-standing vision problem: (1) holistic image training and prediction; and (2) multi-scale and multi-level feature learning. Our proposed method, holistically-nested edge detection (HED), performs image-to-image prediction by means of a deep learning model that leverages fully convolutional neural networks and deeply-supervised nets. HED automatically learns rich hierarchical representations (guided by deep supervision on side responses) that are important in order to approach the human ability resolve the challenging ambiguity in edge and object boundary detection. We significantly advance the state-of-the-art on the BSD500 dataset (ODS F-score of .782) and the NYU Depth dataset (ODS F-score of .746), and do so with an improved speed (0.4 second per image) that is orders of magnitude faster than some recent CNN-based edge detection algorithms.

v2 Add appendix A for updated results (ODS=0.790) on BSDS-500 in a new experiment setting. Fix typos and reorganize formulations. Add Table 2 to discuss the role of deep supervision. Add links to publicly available repository for code, models and data

References in corpus (4)

• Pixel-wise Deep Learning for Contour Detection
• Fast Edge Detection Using Structured Forests
• Oriented Edge Forests for Boundary Detection
• $N^4$-Fields: Neural Network Nearest Neighbor Fields for Image Transforms

Cited by in corpus (117)

• Attention U-Net: Learning Where to Look for the Pancreas
• ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data
• Stacked Hourglass Networks for Human Pose Estimation
• Beyond Skip Connections: Top-Down Modulation for Object Detection
• Recent Advances in Convolutional Neural Networks
• Multimodal Unsupervised Image-to-Image Translation
• Generative Adversarial Network for Medical Images (MI-GAN)
• Abdominal multi-organ segmentation with organ-attention networks and statistical fusion
• Scene Text Detection via Holistic, Multi-Channel Prediction
• Learning Depth from Single Images with Deep Neural Network Embedding Focal Length
• DFR: Deep Feature Reconstruction for Unsupervised Anomaly Segmentation
• Fully automatic detection and segmentation of abdominal aortic thrombus in post-operative CTA images using deep convolutional neural networks
• Deep Contrast Learning for Salient Object Detection
• Improving Deep Pancreas Segmentation in CT and MRI Images via Recurrent Neural Contextual Learning and Direct Loss Function
• BA-Net: Dense Bundle Adjustment Network
• EAST: An Efficient and Accurate Scene Text Detector
• Learning Deep Structured Multi-Scale Features using Attention-Gated CRFs for Contour Prediction
• Deep multi-task learning for a geographically-regularized semantic segmentation of aerial images
• Single-Image Depth Perception in the Wild
• Density-aware Single Image De-raining using a Multi-stream Dense Network
• Learning Representations for Automatic Colorization
• Adversarial Examples for Semantic Segmentation and Object Detection
• Multi-Oriented Text Detection with Fully Convolutional Networks
• Faster Training of Mask R-CNN by Focusing on Instance Boundaries
• Learning to Measure Change: Fully Convolutional Siamese Metric Networks for Scene Change Detection
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• Learning to Refine Object Segments
• 2017 Robotic Instrument Segmentation Challenge
• PixelNet: Towards a General Pixel-level Architecture
• Label Refinement Network for Coarse-to-Fine Semantic Segmentation
• Pushing the Boundaries of Boundary Detection using Deep Learning
• Deep Learning Convolutional Networks for Multiphoton Microscopy Vasculature Segmentation
• Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation
• Detailed Garment Recovery from a Single-View Image
• Image Companding and Inverse Halftoning using Deep Convolutional Neural Networks
• Retinal Vasculature Segmentation Using Local Saliency Maps and Generative Adversarial Networks For Image Super Resolution
• SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis
• Unsupervised learning of object semantic parts from internal states of CNNs by population encoding
• Pixel-wise Attentional Gating for Parsimonious Pixel Labeling
• Liver Segmentation from Multimodal Images using HED-Mask R-CNN
• Generative Adversarial Network-based Synthesis of Visible Faces from Polarimetric Thermal Faces
• Hallucinated-IQA: No-Reference Image Quality Assessment via Adversarial Learning
• Laplacian Pyramid Reconstruction and Refinement for Semantic Segmentation
• Pancreas Segmentation in CT and MRI Images via Domain Specific Network Designing and Recurrent Neural Contextual Learning
• Monocular Depth Estimation using Multi-Scale Continuous CRFs as Sequential Deep Networks
• Real-Time User-Guided Image Colorization with Learned Deep Priors
• Look at Boundary: A Boundary-Aware Face Alignment Algorithm
• Colorful Image Colorization
• Spatial Aggregation of Holistically-Nested Networks for Automated Pancreas Segmentation
• Object Skeleton Extraction in Natural Images by Fusing Scale-associated Deep Side Outputs
• Learning Cross-Modal Deep Representations for Robust Pedestrian Detection
• Label-free segmentation from cardiac ultrasound using self-supervised learning
• EdgeStereo: A Context Integrated Residual Pyramid Network for Stereo Matching
• What is Holding Back Convnets for Detection?
• Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries
• TextMountain: Accurate Scene Text Detection via Instance Segmentation
• Hierarchical Bilinear Pooling for Fine-Grained Visual Recognition
• Deep Embedding Convolutional Neural Network for Synthesizing CT Image from T1-Weighted MR Image
• Visual Anomaly Detection for Images: A Survey
• Spatial Aggregation of Holistically-Nested Convolutional Neural Networks for Automated Pancreas Localization and Segmentation
• 3D convolutional neural network for abdominal aortic aneurysm segmentation
• Learning Anytime Predictions in Neural Networks via Adaptive Loss Balancing
• Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
• SREdgeNet: Edge Enhanced Single Image Super Resolution using Dense Edge Detection Network and Feature Merge Network
• SemiContour: A Semi-supervised Learning Approach for Contour Detection
• Recent Advances in the Applications of Convolutional Neural Networks to Medical Image Contour Detection
• Semantically-Guided Video Object Segmentation
• Self-produced Guidance for Weakly-supervised Object Localization
• Video Object Segmentation using Tracked Object Proposals
• Automatic Vertebra Labeling in Large-Scale 3D CT using Deep Image-to-Image Network with Message Passing and Sparsity Regularization
• Points2Pix: 3D Point-Cloud to Image Translation using conditional Generative Adversarial Networks
• Accelerating Deep Neural Networks with Spatial Bottleneck Modules
• Context-Aware Semantic Inpainting
• Semantic and Contrast-Aware Saliency
• SketchyScene: Richly-Annotated Scene Sketches
• Semantic Road Layout Understanding by Generative Adversarial Inpainting
• LUCSS: Language-based User-customized Colourization of Scene Sketches
• A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection
• Collaborative Layer-wise Discriminative Learning in Deep Neural Networks
• Railway Track Specific Traffic Signal Selection Using Deep Learning
• Doubly Nested Network for Resource-Efficient Inference
• Deep Green Function Convolution for Improving Saliency in Convolutional Neural Networks
• SORT: Second-Order Response Transform for Visual Recognition
• Learning to Segment Human Body Parts with Synthetically Trained Deep Convolutional Networks
• Deeply-Supervised CNN for Prostate Segmentation
• HyperFusion-Net: Densely Reflective Fusion for Salient Object Detection
• Retinal Vessel Segmentation Based on Conditional Deep Convolutional Generative Adversarial Networks
• Contrast-Oriented Deep Neural Networks for Salient Object Detection
• Gradually Updated Neural Networks for Large-Scale Image Recognition
• Continual Occlusions and Optical Flow Estimation
• Cascaded Segmentation-Detection Networks for Word-Level Text Spotting
• Fractal Dimension Invariant Filtering and Its CNN-based Implementation
• Line Artist: A Multiple Style Sketch to Painting Synthesis Scheme
• Point-wise mutual information-based video segmentation with high temporal consistency
• Human Pose Estimation using Global and Local Normalization
• Dense Prediction with Attentive Feature Aggregation
• Multichannel Semantic Segmentation with Unsupervised Domain Adaptation
• Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks
• Object Boundary Guided Semantic Segmentation
• End-to-End Multi-Object Tracking with Global Response Map
• Rethinking Lightweight Convolutional Neural Networks for Efficient and High-quality Pavement Crack Detection
• Machine Learning based Post Processing Artifact Reduction in HEVC Intra Coding
• Saliency Detection via Combining Region-Level and Pixel-Level Predictions with CNNs
• On the Exploration of Convolutional Fusion Networks for Visual Recognition
• Input Fast-Forwarding for Better Deep Learning
• Relative Depth Order Estimation Using Multi-scale Densely Connected Convolutional Networks
• Linear Span Network for Object Skeleton Detection
• SparseMask: Differentiable Connectivity Learning for Dense Image Prediction
• Unsupervised Image-to-Image Translation with Stacked Cycle-Consistent Adversarial Networks
• Deep nested level sets: Fully automated segmentation of cardiac MR images in patients with pulmonary hypertension
• Gland Instance Segmentation by Deep Multichannel Side Supervision
• Learning Pixel-wise Labeling from the Internet without Human Interaction
• Two-stream convolutional neural network for accurate RGB-D fingertip detection using depth and edge information
• Multi-stage Multi-recursive-input Fully Convolutional Networks for Neuronal Boundary Detection
• RGB-D Salient Object Detection Based on Discriminative Cross-modal Transfer Learning
• Multi-Scale Spatially-Asymmetric Recalibration for Image Classification
• W-net: Simultaneous segmentation of multi-anatomical retinal structures using a multi-task deep neural network