ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation

Abstract

The ability to perform pixel-wise semantic segmentation in real-time is of paramount importance in mobile applications. Recent deep neural networks aimed at this task have the disadvantage of requiring a large number of floating point operations and have long run-times that hinder their usability. In this paper, we propose a novel deep neural network architecture named ENet (efficient neural network), created specifically for tasks requiring low latency operation. ENet is up to 18$\times$ faster, requires 75$\times$ less FLOPs, has 79$\times$ less parameters, and provides similar or better accuracy to existing models. We have tested it on CamVid, Cityscapes and SUN datasets and report on comparisons with existing state-of-the-art methods, and the trade-offs between accuracy and processing time of a network. We present performance measurements of the proposed architecture on embedded systems and suggest possible software improvements that could make ENet even faster.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• cuDNN: Efficient Primitives for Deep Learning
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Learning Deconvolution Network for Semantic Segmentation
• Deep Networks with Stochastic Depth
• Flattened Convolutional Neural Networks for Feedforward Acceleration
• Identity Mappings in Deep Residual Networks

Cited by in corpus (234)

• A Survey of Deep Learning Techniques for Autonomous Driving
• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• A Review on Deep Learning Techniques Applied to Semantic Segmentation
• An Analysis of Deep Neural Network Models for Practical Applications
• The ApolloScape Open Dataset for Autonomous Driving and its Application
• Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review
• YOLACT++: Better Real-time Instance Segmentation
• Semantic Instance Segmentation with a Discriminative Loss Function
• Fast-SCNN: Fast Semantic Segmentation Network
• Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey
• NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
• Constrained-CNN losses for weakly supervised segmentation
• nnFormer: Interleaved Transformer for Volumetric Segmentation
• Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes
• Embedding Structured Contour and Location Prior in Siamesed Fully Convolutional Networks for Road Detection
• W-Net: A Deep Model for Fully Unsupervised Image Segmentation
• ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-time
• DISK: Learning local features with policy gradient
• DABNet: Depth-wise Asymmetric Bottleneck for Real-time Semantic Segmentation
• MobileSal: Extremely Efficient RGB-D Salient Object Detection
• Semantic Segmentation of Pathological Lung Tissue with Dilated Fully Convolutional Networks
• Restricted Deformable Convolution based Road Scene Semantic Segmentation Using Surround View Cameras
• ICNet for Real-Time Semantic Segmentation on High-Resolution Images
• BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation
• Estimating Depth from RGB and Sparse Sensing
• BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation
• FasterSeg: Searching for Faster Real-time Semantic Segmentation
• Light-Weight RefineNet for Real-Time Semantic Segmentation
• Automated Evaluation of Semantic Segmentation Robustness for Autonomous Driving
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• Learning Lightweight Lane Detection CNNs by Self Attention Distillation
• Double Similarity Distillation for Semantic Image Segmentation
• ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks
• CGNet: A Light-weight Context Guided Network for Semantic Segmentation
• LiteSeg: A Novel Lightweight ConvNet for Semantic Segmentation
• Curriculum semi-supervised segmentation
• Self-paced and self-consistent co-training for semi-supervised image segmentation
• How much real data do we actually need: Analyzing object detection performance using synthetic and real data
• SLSNet: Skin lesion segmentation using a lightweight generative adversarial network
• Key Points Estimation and Point Instance Segmentation Approach for Lane Detection
• Constrained Deep Networks: Lagrangian Optimization via Log-Barrier Extensions
• Improving Fast Segmentation With Teacher-student Learning
• DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation
• LeViT-UNet: Make Faster Encoders with Transformer for Medical Image Segmentation
• Driving Scene Perception Network: Real-time Joint Detection, Depth Estimation and Semantic Segmentation
• Gated CRF Loss for Weakly Supervised Semantic Image Segmentation
• Real-time Semantic Image Segmentation via Spatial Sparsity
• Fast Scene Understanding for Autonomous Driving
• ATISS: Autoregressive Transformers for Indoor Scene Synthesis
• ShuffleSeg: Real-time Semantic Segmentation Network
• Real-Time Semantic Segmentation via Multiply Spatial Fusion Network
• TNCR: Table Net Detection and Classification Dataset
• MiniSeg: An Extremely Minimum Network for Efficient COVID-19 Segmentation
• Multi-Modal Obstacle Detection in Unstructured Environments with Conditional Random Fields
• Guided Upsampling Network for Real-Time Semantic Segmentation
• Bounding Box Tightness Prior for Weakly Supervised Image Segmentation
• The Lovász-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks
• Rethinking BiSeNet For Real-time Semantic Segmentation
• PDNet: Semantic Segmentation integrated with a Primal-Dual Network for Document binarization
• Learning the Pareto Front with Hypernetworks
• Constrained domain adaptation for Image segmentation
• Instance Segmentation by Deep Coloring
• Multi-Task Handwritten Document Layout Analysis
• RESA: Recurrent Feature-Shift Aggregator for Lane Detection
• Efficient Dense Modules of Asymmetric Convolution for Real-Time Semantic Segmentation
• VarGNet: Variable Group Convolutional Neural Network for Efficient Embedded Computing
• Adapting Semantic Segmentation Models for Changes in Illumination and Camera Perspective
• An efficient solution for semantic segmentation: ShuffleNet V2 with atrous separable convolutions
• Agnostic Lane Detection
• GASNet: Weakly-supervised Framework for COVID-19 Lesion Segmentation
• On the effectiveness of GAN generated cardiac MRIs for segmentation
• Residual Pyramid Learning for Single-Shot Semantic Segmentation
• LEDNet: A Lightweight Encoder-Decoder Network for Real-Time Semantic Segmentation
• Knowledge Adaptation for Efficient Semantic Segmentation
• 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans
• Feature Pyramid Encoding Network for Real-time Semantic Segmentation
• Label-guided Attention Distillation for Lane Segmentation
• ExtremeC3Net: Extreme Lightweight Portrait Segmentation Networks using Advanced C3-modules
• A survey of Object Classification and Detection based on 2D/3D data
• Efficient semantic image segmentation with superpixel pooling
• Efficient Accelerator for Dilated and Transposed Convolution with Decomposition
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Boundary-Aware Network for Fast and High-Accuracy Portrait Segmentation
• Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes
• Active Learning for Deep Detection Neural Networks
• Bounding boxes for weakly supervised segmentation: Global constraints get close to full supervision
• CFPNet-M: A Light-Weight Encoder-Decoder Based Network for Multimodal Biomedical Image Real-Time Segmentation
• Structured Knowledge Distillation for Dense Prediction
• fpgaConvNet: A Toolflow for Mapping Diverse Convolutional Neural Networks on Embedded FPGAs
• Semantic Flow for Fast and Accurate Scene Parsing
• C3: Concentrated-Comprehensive Convolution and its application to semantic segmentation
• Pixel Difference Networks for Efficient Edge Detection
• Chargrid: Towards Understanding 2D Documents
• Multi-source Pseudo-label Learning of Semantic Segmentation for the Scene Recognition of Agricultural Mobile Robots
• Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer
• Customizable Architecture Search for Semantic Segmentation
• Real-Time High-Performance Semantic Image Segmentation of Urban Street Scenes
• Highly Efficient Salient Object Detection with 100K Parameters
• Lite-HRNet: A Lightweight High-Resolution Network
• Recent Advances in the Applications of Convolutional Neural Networks to Medical Image Contour Detection
• A Survey on Deep Learning Methods for Semantic Image Segmentation in Real-Time
• VolMap: A Real-time Model for Semantic Segmentation of a LiDAR surrounding view
• Temporally Distributed Networks for Fast Video Semantic Segmentation
• Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search
• Real-time Progressive 3D Semantic Segmentation for Indoor Scene
• One Versus all for deep Neural Network Incertitude (OVNNI) quantification
• Efficient Road Lane Marking Detection with Deep Learning
• Inter-Region Affinity Distillation for Road Marking Segmentation
• Real-time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-driving Images
• Efficient Semantic Video Segmentation with Per-frame Inference
• Building an Integrated Mobile Robotic System for Real-Time Applications in Construction
• DeepPerimeter: Indoor Boundary Estimation from Posed Monocular Sequences
• MSCFNet: A Lightweight Network With Multi-Scale Context Fusion for Real-Time Semantic Segmentation
• Training Bit Fully Convolutional Network for Fast Semantic Segmentation
• Points2Pix: 3D Point-Cloud to Image Translation using conditional Generative Adversarial Networks
• DS-PASS: Detail-Sensitive Panoramic Annular Semantic Segmentation through SwaftNet for Surrounding Sensing
• Automatic breast cancer grading in lymph nodes using a deep neural network
• FisheyeMultiNet: Real-time Multi-task Learning Architecture for Surround-view Automated Parking System
• LRNNet: A Light-Weighted Network with Efficient Reduced Non-Local Operation for Real-Time Semantic Segmentation
• Reinforced Wasserstein Training for Severity-Aware Semantic Segmentation in Autonomous Driving
• RevealNet: Seeing Behind Objects in RGB-D Scans
• An Intelligent Safety System for Human-Centered Semi-Autonomous Vehicles
• Real-world Mapping of Gaze Fixations Using Instance Segmentation for Road Construction Safety Applications
• Accelerating Large-Kernel Convolution Using Summed-Area Tables
• Fast Face-swap Using Convolutional Neural Networks
• Graph-guided Architecture Search for Real-time Semantic Segmentation
• Error Correction for Dense Semantic Image Labeling
• DSNet: An Efficient CNN for Road Scene Segmentation
• Fast Deep Matting for Portrait Animation on Mobile Phone
• Artificial intelligence based prediction on lung cancer risk factors using deep learning
• DeLS-3D: Deep Localization and Segmentation with a 3D Semantic Map
• Real-time Semantic Segmentation with Fast Attention
• Semantic Road Layout Understanding by Generative Adversarial Inpainting
• Auxiliary Learning for Deep Multi-task Learning
• Trained Trajectory based Automated Parking System using Visual SLAM on Surround View Cameras
• SymmNet: A Symmetric Convolutional Neural Network for Occlusion Detection
• Empirical Study of Multi-Task Hourglass Model for Semantic Segmentation Task
• 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation
• 3D-to-2D Distillation for Indoor Scene Parsing
• Background Subtraction with Real-time Semantic Segmentation
• Function Norms and Regularization in Deep Networks
• V-FCNN: Volumetric Fully Convolution Neural Network For Automatic Atrial Segmentation
• Deep Learning-based Aerial Image Segmentation with Open Data for Disaster Impact Assessment
• CFPNet: Channel-wise Feature Pyramid for Real-Time Semantic Segmentation
• A Large-Scale Dataset for Benchmarking Elevator Button Segmentation and Character Recognition
• Towards Real-Time Automatic Portrait Matting on Mobile Devices
• An Adaptive Approach for Automated Grapevine Phenotyping using VGG-based Convolutional Neural Networks
• ESNet: An Efficient Symmetric Network for Real-time Semantic Segmentation
• Design and Development of Autonomous Delivery Robot
• On Machine Learning and Structure for Mobile Robots
• FDDWNet: A Lightweight Convolutional Neural Network for Real-time Sementic Segmentation
• Predicting Terrain Mechanical Properties in Sight for Planetary Rovers with Semantic Clues
• Rethinking Convolutional Semantic Segmentation Learning
• ASL Recognition with Metric-Learning based Lightweight Network
• Low-Latency Video Semantic Segmentation
• A Methodological Review of Visual Road Recognition Procedures for Autonomous Driving Applications
• See Clearer at Night: Towards Robust Nighttime Semantic Segmentation through Day-Night Image Conversion
• Adversarial Multiscale Feature Learning for Overlapping Chromosome Segmentation
• FarSee-Net: Real-Time Semantic Segmentation by Efficient Multi-scale Context Aggregation and Feature Space Super-resolution
• Incorporating Luminance, Depth and Color Information by a Fusion-based Network for Semantic Segmentation
• A Compact DNN: Approaching GoogLeNet-Level Accuracy of Classification and Domain Adaptation
• ShelfNet for Fast Semantic Segmentation
• HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
• Real-time Multi-target Path Prediction and Planning for Autonomous Driving aided by FCN
• Multi-Task Learning for Automotive Foggy Scene Understanding via Domain Adaptation to an Illumination-Invariant Representation
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• Lane detection in complex scenes based on end-to-end neural network
• Motion Planning for Heterogeneous Unmanned Systems under Partial Observation from UAV
• Automatic Real-time Background Cut for Portrait Videos
• Efficient Semantic Segmentation using Gradual Grouping
• MODS -- A USV-oriented object detection and obstacle segmentation benchmark
• SINet: Extreme Lightweight Portrait Segmentation Networks with Spatial Squeeze Modules and Information Blocking Decoder
• Discretely-constrained deep network for weakly supervised segmentation
• Exploiting Activation based Gradient Output Sparsity to Accelerate Backpropagation in CNNs
• AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training
• Exploring the Effects of Data Augmentation for Drivable Area Segmentation
• Importance-Aware Semantic Segmentation with Efficient Pyramidal Context Network for Navigational Assistant Systems
• Generating superpixels using deep image representations
• EADNet: Efficient Asymmetric Dilated Network for Semantic Segmentation
• ESFNet: Efficient Network for Building Extraction from High-Resolution Aerial Images
• Implicit Integration of Superpixel Segmentation into Fully Convolutional Networks
• Real-Time Semantic Segmentation via Auto Depth, Downsampling Joint Decision and Feature Aggregation
• Boundary Guidance Hierarchical Network for Real-Time Tongue Segmentation
• MT: Multi-Perspective Feature Learning Network for Scene Text Detection
• Neuromodulated Patience for Robot and Self-Driving Vehicle Navigation
• Adaptive Class Weight based Dual Focal Loss for Improved Semantic Segmentation
• Superpixel-Guided Label Softening for Medical Image Segmentation
• FastVentricle: Cardiac Segmentation with ENet
• Collaborative Global-Local Networks for Memory-Efficient Segmentation of Ultra-High Resolution Images
• Token Pooling in Vision Transformers
• AutoLay: Benchmarking amodal layout estimation for autonomous driving
• What Synthesis is Missing: Depth Adaptation Integrated with Weak Supervision for Indoor Scene Parsing
• Online Self-supervised Scene Segmentation for Micro Aerial Vehicles
• DFNet: Semantic Segmentation on Panoramic Images with Dynamic Loss Weights and Residual Fusion Block
• Fast and Accurate Scene Parsing via Bi-direction Alignment Networks
• Associative Embedding for Game-Agnostic Team Discrimination
• Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real World
• Global Context Networks
• Capturing Omni-Range Context for Omnidirectional Segmentation
• Importance-Aware Semantic Segmentation in Self-Driving with Discrete Wasserstein Training
• NoFADE: Analyzing Diminishing Returns on CO2 Investment
• Investigations on the inference optimization techniques and their impact on multiple hardware platforms for Semantic Segmentation
• Rethinking Lightweight Convolutional Neural Networks for Efficient and High-quality Pavement Crack Detection
• LORCK: Learnable Object-Resembling Convolution Kernels
• TPPI-Net: Towards Efficient and Practical Hyperspectral Image Classification
• Beyond Single Stage Encoder-Decoder Networks: Deep Decoders for Semantic Image Segmentation
• A Comparative Study of High-Recall Real-Time Semantic Segmentation Based on Swift Factorized Network
• Dense Dual-Path Network for Real-time Semantic Segmentation
• AASeg: Attention Aware Network for Real Time Semantic Segmentation
• CoCo DistillNet: a Cross-layer Correlation Distillation Network for Pathological Gastric Cancer Segmentation
• LEAN: graph-based pruning for convolutional neural networks by extracting longest chains
• MyFood: A Food Segmentation and Classification System to Aid Nutritional Monitoring
• Learning-based Tracking of Fast Moving Objects
• Spatial Sampling Network for Fast Scene Understanding
• Fast Portrait Segmentation with Highly Light-weight Network
• Compact retail shelf segmentation for mobile deployment
• Safety Metrics for Semantic Segmentation in Autonomous Driving
• Stochastic Function Norm Regularization of Deep Networks
• ItNet: iterative neural networks with small graphs for accurate, efficient and anytime semantic segmentation
• Left ventricle segmentation By modelling uncertainty in prediction of deep convolutional neural networks and adaptive thresholding inference
• Deep Control - a simple automatic gain control for memory efficient and high performance training of deep convolutional neural networks
• Hybrid tracker based optimal path tracking system for complex road environments for autonomous driving
• Multi-Model Learning for Real-Time Automotive Semantic Foggy Scene Understanding via Domain Adaptation
• TapLab: A Fast Framework for Semantic Video Segmentation Tapping into Compressed-Domain Knowledge
• Context-aware virtual adversarial training for anatomically-plausible segmentation
• WeedMap: A large-scale semantic weed mapping framework using aerial multispectral imaging and deep neural network for precision farming
• Digital Editions as Distant Supervision for Layout Analysis of Printed Books
• Semantic sensor fusion: from camera to sparse lidar information
• A De-raining semantic segmentation network for real-time foreground segmentation
• Multi-lane Detection Using Instance Segmentation and Attentive Voting
• Aerial-PASS: Panoramic Annular Scene Segmentation in Drone Videos
• Efficient Semantic Segmentation for Visual Bird's-eye View Interpretation
• A Dataset of Laryngeal Endoscopic Images with Comparative Study on Convolution Neural Network Based Semantic Segmentation
• Latent Transformations for Object View Points Synthesis