PixelNet: Representation of the pixels, by the pixels, and for the pixels

Abstract

We explore design principles for general pixel-level prediction problems, from low-level edge detection to mid-level surface normal estimation to high-level semantic segmentation. Convolutional predictors, such as the fully-convolutional network (FCN), have achieved remarkable success by exploiting the spatial redundancy of neighboring pixels through convolutional processing. Though computationally efficient, we point out that such approaches are not statistically efficient during learning precisely because spatial redundancy limits the information learned from neighboring pixels. We demonstrate that stratified sampling of pixels allows one to (1) add diversity during batch updates, speeding up learning; (2) explore complex nonlinear predictors, improving accuracy; and (3) efficiently train state-of-the-art models tabula rasa (i.e., "from scratch") for diverse pixel-labeling tasks. Our single architecture produces state-of-the-art results for semantic segmentation on PASCAL-Context dataset, surface normal estimation on NYUDv2 depth dataset, and edge detection on BSDS.

arXiv commentProject Page: http://www.cs.cmu.edu/~aayushb/pixelNet/. arXiv admin note: substantial text overlap with arXiv:1609.06694

References in corpus (30)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Multi-Scale Context Aggregation by Dilated Convolutions
• ParseNet: Looking Wider to See Better
• Fully Convolutional Networks for Semantic Segmentation
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Learning Deconvolution Network for Semantic Segmentation
• FlowNet: Learning Optical Flow with Convolutional Networks
• DenseBox: Unifying Landmark Localization with End to End Object Detection
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• Convolutional Pose Machines
• Unsupervised Visual Representation Learning by Context Prediction
• Unsupervised Learning of Visual Representations using Videos
• Learning to See by Moving
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
• An Implementation of Faster RCNN with Study for Region Sampling
• Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• Learning Representations for Automatic Colorization
• Pixel-wise Deep Learning for Contour Detection
• Recurrent Convolutional Neural Networks for Scene Parsing
• Analyzing the Performance of Multilayer Neural Networks for Object Recognition
• Feedforward semantic segmentation with zoom-out features
• Designing Deep Networks for Surface Normal Estimation
• Learning to Extract Motion from Videos in Convolutional Neural Networks
• Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
• Unsupervised Learning of Spatiotemporally Coherent Metrics
• Top-Down Learning for Structured Labeling with Convolutional Pseudoprior
• Oriented Edge Forests for Boundary Detection

Cited by in corpus (26)

• Multi-Task Learning for Dense Prediction Tasks: A Survey
• The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches
• PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
• Adversarial Networks for Spatial Context-Aware Spectral Image Reconstruction from RGB
• GeoNet++: Iterative Geometric Neural Network with Edge-Aware Refinement for Joint Depth and Surface Normal Estimation
• Hierarchical Autoregressive Image Models with Auxiliary Decoders
• Lucid Data Dreaming for Video Object Segmentation
• A Multi-Stage Multi-Task Neural Network for Aerial Scene Interpretation and Geolocalization
• Dense Recurrent Neural Networks for Scene Labeling
• PixelNN: Example-based Image Synthesis
• Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery
• Video Representation Learning and Latent Concept Mining for Large-scale Multi-label Video Classification
• DivergentNets: Medical Image Segmentation by Network Ensemble
• Patch Correspondences for Interpreting Pixel-level CNNs
• Streaming Self-Training via Domain-Agnostic Unlabeled Images
• Explainable Semantic Mapping for First Responders
• Weakly- and Semi-Supervised Panoptic Segmentation
• Scene Parsing via Dense Recurrent Neural Networks with Attentional Selection
• Weakly-Supervised Cell Tracking via Backward-and-Forward Propagation
• Taming Adversarial Domain Transfer with Structural Constraints for Image Enhancement
• Conditional Generative Modeling via Learning the Latent Space
• Improving task-specific representation via 1M unlabelled images without any extra knowledge
• Joint Facade Registration and Segmentation for Urban Localization
• Glioma Prognosis: Segmentation of the Tumor and Survival Prediction using Shape, Geometric and Clinical Information
• Superpixel-based Semantic Segmentation Trained by Statistical Process Control
• Sparse2Dense: From direct sparse odometry to dense 3D reconstruction