An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

Abstract

Few ideas have enjoyed as large an impact on deep learning as convolution. For any problem involving pixels or spatial representations, common intuition holds that convolutional neural networks may be appropriate. In this paper we show a striking counterexample to this intuition via the seemingly trivial coordinate transform problem, which simply requires learning a mapping between coordinates in (x,y) Cartesian space and one-hot pixel space. Although convolutional networks would seem appropriate for this task, we show that they fail spectacularly. We demonstrate and carefully analyze the failure first on a toy problem, at which point a simple fix becomes obvious. We call this solution CoordConv, which works by giving convolution access to its own input coordinates through the use of extra coordinate channels. Without sacrificing the computational and parametric efficiency of ordinary convolution, CoordConv allows networks to learn either complete translation invariance or varying degrees of translation dependence, as required by the end task. CoordConv solves the coordinate transform problem with perfect generalization and 150 times faster with 10--100 times fewer parameters than convolution. This stark contrast raises the question: to what extent has this inability of convolution persisted insidiously inside other tasks, subtly hampering performance from within? A complete answer to this question will require further investigation, but we show preliminary evidence that swapping convolution for CoordConv can improve models on a diverse set of tasks. Using CoordConv in a GAN produced less mode collapse as the transform between high-level spatial latents and pixels becomes easier to learn. A Faster R-CNN detection model trained on MNIST showed 24% better IOU when using CoordConv, and in the RL domain agents playing Atari games benefit significantly from the use of CoordConv layers.

arXiv commentPublished in NeurIPS 2018

References in corpus (10)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• WaveNet: A Generative Model for Raw Audio
• Convolutional Sequence to Sequence Learning
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• DRAW: A Recurrent Neural Network For Image Generation
• Deep Image Prior
• A simple neural network module for relational reasoning
• Distributed Prioritized Experience Replay
• Measuring the Intrinsic Dimension of Objective Landscapes
• Road Segmentation Using CNN with GRU

Cited by in corpus (17)

• A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS
• STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos
• Soil Texture Classification with 1D Convolutional Neural Networks based on Hyperspectral Data
• CoherentGS: Sparse Novel View Synthesis with Coherent 3D Gaussians
• AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation
• Rethinking Closed-loop Training for Autonomous Driving
• AgingMapGAN (AMGAN): High-Resolution Controllable Face Aging with Spatially-Aware Conditional GANs
• Automated Left Ventricle Dimension Measurement in 2D Cardiac Ultrasound via an Anatomically Meaningful CNN Approach
• Unsupervised Image to Sequence Translation with Canvas-Drawer Networks
• The particle track reconstruction based on deep learning neural networks
• Learning from similarity and information extraction from structured documents
• Model-free estimation of the Cramér-Rao bound for deep-learning microscopy in complex media
• Domain-informed neural networks for interaction localization within astroparticle experiments
• Deep Multicameral Decoding for Localizing Unoccluded Object Instances from a Single RGB Image
• Table understanding in structured documents
• End to End Face Reconstruction via Differentiable PnP
• Nondiscriminatory Treatment: a straightforward framework for multi-human parsing