DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
arXiv:1606.00915
Abstract
In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First, we highlight convolution with upsampled filters, or 'atrous convolution', as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second, we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third, we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed "DeepLab" system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7% mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.
arXiv commentAccepted by TPAMI
References in corpus (31)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
- ParseNet: Looking Wider to See Better
- Learning Deconvolution Network for Semantic Segmentation
- Convolutional Feature Masking for Joint Object and Stuff Segmentation
- Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation
- ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
- Fully Connected Deep Structured Networks
- Identity Mappings in Deep Residual Networks
- Semantic Image Segmentation via Deep Parsing Network
- Bridging Category-level and Instance-level Semantic Image Segmentation
- BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation
- Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
- Proposal-free Network for Instance-level Object Segmentation
- High-performance Semantic Segmentation Using Very Deep Fully Convolutional Networks
- Instance-sensitive Fully Convolutional Networks
- Fast Image Scanning with Deep Max-Pooling Convolutional Neural Networks
- Combining the Best of Convolutional Layers and Recurrent Layers: A Hybrid Network for Semantic Segmentation
- Pixel-level Encoding and Depth Layering for Instance-level Semantic Labeling
- Laplacian Pyramid Reconstruction and Refinement for Semantic Segmentation
- Semantic Object Parsing with Local-Global Long Short-Term Memory
- Joint Object and Part Segmentation using Deep Learned Potentials
- Material Recognition in the Wild with the Materials in Context Database
- Optical Flow with Semantic Segmentation and Localized Layers
- Combining the Best of Graphical Models and ConvNets for Semantic Segmentation
- Zoom Better to See Clearer: Human and Object Parsing with Hierarchical Auto-Zoom Net
- From Image-level to Pixel-level Labeling with Convolutional Networks
- Fast Semantic Image Segmentation with High Order Context and Guided Filtering
- Semantic Object Parsing with Graph LSTM
Cited by in corpus (206)
- Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data
- Momentum Contrast for Unsupervised Visual Representation Learning
- SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
- A deep learning model integrating FCNNs and CRFs for brain tumor segmentation
- Learning Spectral-Spatial-Temporal Features via a Recurrent Convolutional Neural Network for Change Detection in Multispectral Imagery
- Rethinking Pre-training and Self-training
- Adversarial Learning for Semi-Supervised Semantic Segmentation
- On the Compactness, Efficiency, and Representation of 3D Convolutional Networks: Brain Parcellation as a Pretext Task
- Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey
- Pyramid Scene Parsing Network
- Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
- FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation
- Convolutional neural networks automate detection for tracking of submicron scale particles in 2D and 3D
- Semantic Understanding of Scenes through the ADE20K Dataset
- Dilated Residual Networks
- Robust Spatial Filtering with Graph Convolutional Neural Networks
- Virtual to Real Reinforcement Learning for Autonomous Driving
- Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
- Examining the Impact of Blur on Recognition by Convolutional Networks
- Context Encoding for Semantic Segmentation
- ICNet for Real-Time Semantic Segmentation on High-Resolution Images
- Image Segmentation Algorithms Overview
- BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation
- Automated sub-cortical brain structure segmentation combining spatial and deep convolutional features
- Light-Weight RefineNet for Real-Time Semantic Segmentation
- B-CNN: Branch Convolutional Neural Network for Hierarchical Classification
- Towards Image Understanding from Deep Compression without Decoding
- What Do We Understand About Convolutional Networks?
- PixelNet: Representation of the pixels, by the pixels, and for the pixels
- Adversarial Examples for Semantic Segmentation and Object Detection
- Learning a Discriminative Feature Network for Semantic Segmentation
- Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing
- Towards Accurate Multi-person Pose Estimation in the Wild
- Gated-Dilated Networks for Lung Nodule Classification in CT scans
- Learning Video Object Segmentation from Static Images
- The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation
- Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
- RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation
- CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos
- PixelNet: Towards a General Pixel-level Architecture
- Driving Scene Perception Network: Real-time Joint Detection, Depth Estimation and Semantic Segmentation
- Real-time Semantic Image Segmentation via Spatial Sparsity
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
- A Survey on Deep Learning Methods for Robot Vision
- Predicting Scene Parsing and Motion Dynamics in the Future
- Looking at Outfit to Parse Clothing
- Semi-Dense 3D Semantic Mapping from Monocular SLAM
- CUTIE: Learning to Understand Documents with Convolutional Universal Text Information Extractor
- Visual Affordance and Function Understanding: A Survey
- Learning to Branch for Multi-Task Learning
- SRM : A Style-based Recalibration Module for Convolutional Neural Networks
- Domain Adaptation for Semantic Segmentation with Maximum Squares Loss
- The Lovász-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks
- ResizeMix: Mixing Data with Preserved Object Information and True Labels
- DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
- Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization
- MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features
- Interactive Video Object Segmentation in the Wild
- Scene Text Synthesis for Efficient and Effective Deep Network Training
- All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
- Exploiting saliency for object segmentation from image level labels
- Simple Does It: Weakly Supervised Instance and Semantic Segmentation
- Pixel Deconvolutional Networks
- Convolutional Neural Pyramid for Image Processing
- Getting to 99% Accuracy in Interactive Segmentation
- FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation
- Learning Affinity via Spatial Propagation Networks
- Learning Implicitly Recurrent CNNs Through Parameter Sharing
- Not All Pixels Are Equal: Difficulty-aware Semantic Segmentation via Deep Layer Cascade
- Deep Learning for Pancreas Segmentation: a Systematic Review
- Multi-Scale Coarse-to-Fine Segmentation for Screening Pancreatic Ductal Adenocarcinoma
- Fashioning with Networks: Neural Style Transfer to Design Clothes
- DCNAS: Densely Connected Neural Architecture Search for Semantic Image Segmentation
- Macroscale fracture surface segmentation via semi-supervised learning considering the structural similarity
- On Regularized Losses for Weakly-supervised CNN Segmentation
- CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images
- All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation
- Exploit fully automatic low-level segmented PET data for training high-level deep learning algorithms for the corresponding CT data
- Cost Volume Pyramid Based Depth Inference for Multi-View Stereo
- Convolutional Video Steganography with Temporal Residual Modeling
- Counterfactual Critic Multi-Agent Training for Scene Graph Generation
- Efficient Accelerator for Dilated and Transposed Convolution with Decomposition
- Mixup-CAM: Weakly-supervised Semantic Segmentation via Uncertainty Regularization
- Blazingly Fast Video Object Segmentation with Pixel-Wise Metric Learning
- CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement
- Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes
- Joint Graph Decomposition and Node Labeling: Problem, Algorithms, Applications
- Scene Parsing with Global Context Embedding
- Correlation Maximized Structural Similarity Loss for Semantic Segmentation
- A Relation-Augmented Fully Convolutional Network for Semantic Segmentation in Aerial Scenes
- Object Detection Free Instance Segmentation With Labeling Transformations
- Star Shape Prior in Fully Convolutional Networks for Skin Lesion Segmentation
- Deep Extreme Cut: From Extreme Points to Object Segmentation
- Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer
- Learning Dynamic Routing for Semantic Segmentation
- Cross-domain Human Parsing via Adversarial Feature and Label Adaptation
- Learning Deep Representations for Semantic Image Parsing: a Comprehensive Overview
- Fast, Exact and Multi-Scale Inference for Semantic Image Segmentation with Deep Gaussian CRFs
- Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
- An Embarrassingly Simple Approach for Knowledge Distillation
- Pointwise Convolutional Neural Networks
- Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast
- Actor-Action Semantic Segmentation with Region Masks
- Weakly-Supervised Semantic Segmentation via Sub-category Exploration
- FoveaNet: Perspective-aware Urban Scene Parsing
- MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens
- CO2: Consistent Contrast for Unsupervised Visual Representation Learning
- Weakly Supervised Semantic Segmentation Based on Web Image Co-segmentation
- Look Before You Leap: Learning Landmark Features for One-Stage Visual Grounding
- Zero-Shot Semantic Segmentation
- AdaCoSeg: Adaptive Shape Co-Segmentation with Group Consistency Loss
- STEP: Segmenting and Tracking Every Pixel
- Relating Input Concepts to Convolutional Neural Network Decisions
- LiftPool: Bidirectional ConvNet Pooling
- Super-Resolution with Deep Adaptive Image Resampling
- Explorations and Lessons Learned in Building an Autonomous Formula SAE Car from Simulations
- Referring Expression Object Segmentation with Caption-Aware Consistency
- Accelerating Deep Neural Networks with Spatial Bottleneck Modules
- Rapid Exact Signal Scanning with Deep Convolutional Neural Networks
- Weakly and Semi Supervised Human Body Part Parsing via Pose-Guided Knowledge Transfer
- Dense Recurrent Neural Networks for Scene Labeling
- A 3D Coarse-to-Fine Framework for Volumetric Medical Image Segmentation
- Bi-Mix: Bidirectional Mixing for Domain Adaptive Nighttime Semantic Segmentation
- SketchyScene: Richly-Annotated Scene Sketches
- A deep learning-based method for prostate segmentation in T2-weighted magnetic resonance imaging
- RefineMask: Towards High-Quality Instance Segmentation with Fine-Grained Features
- Sparsely Aggregated Convolutional Networks
- Learning Multi-modal Information for Robust Light Field Depth Estimation
- Beyond Forward Shortcuts: Fully Convolutional Master-Slave Networks (MSNets) with Backward Skip Connections for Semantic Segmentation
- Adaptive Binarization for Weakly Supervised Affordance Segmentation
- Sketch2code: Generating a website from a paper mockup
- BusyHands: A Hand-Tool Interaction Database for Assembly Tasks Semantic Segmentation
- Boundary-sensitive Network for Portrait Segmentation
- Multigrid Neural Architectures
- Unsupervised domain adaptation via coarse-to-fine feature alignment method using contrastive learning
- The Resistance to Label Noise in K-NN and DNN Depends on its Concentration
- LUCSS: Language-based User-customized Colourization of Scene Sketches
- Dual Encoder Fusion U-Net (DEFU-Net) for Cross-manufacturer Chest X-ray Segmentation
- Outline Objects using Deep Reinforcement Learning
- A Unified Efficient Pyramid Transformer for Semantic Segmentation
- Stacked Neural Networks for end-to-end ciliary motion analysis
- Unsupervised Domain Adaptation for Video Semantic Segmentation
- SeGAN: Segmenting and Generating the Invisible
- EfficientHRNet: Efficient Scaling for Lightweight High-Resolution Multi-Person Pose Estimation
- Improving the Resolution of CNN Feature Maps Efficiently with Multisampling
- Learning Multiple Dense Prediction Tasks from Partially Annotated Data
- MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems
- Influence of Image Classification Accuracy on Saliency Map Estimation
- Rethinking Convolutional Semantic Segmentation Learning
- FaceShapeGene: A Disentangled Shape Representation for Flexible Face Image Editing
- Binge Watching: Scaling Affordance Learning from Sitcoms
- Spatial Memory for Context Reasoning in Object Detection
- Ray-ONet: Efficient 3D Reconstruction From A Single RGB Image
- AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks
- Image Labeling with Markov Random Fields and Conditional Random Fields
- Face Aging with Contextual Generative Adversarial Nets
- Referring Image Segmentation via Cross-Modal Progressive Comprehension
- Gotta Adapt 'Em All: Joint Pixel and Feature-Level Domain Adaptation for Recognition in the Wild
- Improving Augmentation and Evaluation Schemes for Semantic Image Synthesis
- Representative Graph Neural Network
- Inter-BMV: Interpolation with Block Motion Vectors for Fast Semantic Segmentation on Video
- Quality-Aware Network for Human Parsing
- Spatially-Adaptive Filter Units for Deep Neural Networks
- IMP: Instance Mask Projection for High Accuracy Semantic Segmentation of Things
- Parsing R-CNN for Instance-Level Human Analysis
- Patchwork: A Patch-wise Attention Network for Efficient Object Detection and Segmentation in Video Streams
- S4-Net: Geometry-Consistent Semi-Supervised Semantic Segmentation
- Structured 2D Representation of 3D Data for Shape Processing
- Recurrent Multimodal Interaction for Referring Image Segmentation
- Universal Perceptual Grouping
- PointINS: Point-based Instance Segmentation
- AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop
- Image Synthesis via Semantic Composition
- Context Prior for Scene Segmentation
- Deep, Dense, and Low-Rank Gaussian Conditional Random Fields
- Scene Parsing via Dense Recurrent Neural Networks with Attentional Selection
- What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions
- Dilated Spatial Generative Adversarial Networks for Ergodic Image Generation
- Detection and Classification of Breast Cancer Metastates Based on U-Net
- The Ethical Dilemma when (not) Setting up Cost-based Decision Rules in Semantic Segmentation
- Unseen Object Segmentation in Videos via Transferable Representations
- Diagnostics in Semantic Segmentation
- Beyond Planar Symmetry: Modeling human perception of reflection and rotation symmetries in the wild
- Design Pseudo Ground Truth with Motion Cue for Unsupervised Video Object Segmentation
- Unsupervised Video Object Segmentation with Distractor-Aware Online Adaptation
- Semantic See-Through Rendering on Light Fields
- Learning Rich Representations For Structured Visual Prediction Tasks
- RethNet: Object-by-Object Learning for Detecting Facial Skin Problems
- DV3+HED+: A DCNNs-based Framework to Monitor Temporary Works and ESAs in Railway Construction Project Using VHR Satellite Images
- CNN in MRF: Video Object Segmentation via Inference in A CNN-Based Higher-Order Spatio-Temporal MRF
- Quality-Aware Network for Face Parsing
- Multi-Scale Spatially-Asymmetric Recalibration for Image Classification
- Addressing the Invisible: Street Address Generation for Developing Countries with Deep Learning
- Guided Feature Selection for Deep Visual Odometry
- Using Cross-Model EgoSupervision to Learn Cooperative Basketball Intention
- Beyond Gradient Descent for Regularized Segmentation Losses
- 3D Scene Parsing via Class-Wise Adaptation
- Per-Pixel Feedback for improving Semantic Segmentation
- Shelf-Supervised Mesh Prediction in the Wild
- A Distraction Score for Watermarks
- A Study on Trees's Knots Prediction from their Bark Outer-Shape
- MASON: A Model AgnoStic ObjectNess Framework
- A Survey On 3D Inner Structure Prediction from its Outer Shape
- WeedMap: A large-scale semantic weed mapping framework using aerial multispectral imaging and deep neural network for precision farming