Streaming convolutional neural networks for end-to-end learning with multi-megapixel images

Abstract

Due to memory constraints on current hardware, most convolution neural networks (CNN) are trained on sub-megapixel images. For example, most popular datasets in computer vision contain images much less than a megapixel in size (0.09MP for ImageNet and 0.001MP for CIFAR-10). In some domains such as medical imaging, multi-megapixel images are needed to identify the presence of disease accurately. We propose a novel method to directly train convolutional neural networks using any input image size end-to-end. This method exploits the locality of most operations in modern convolutional neural networks by performing the forward and backward pass on smaller tiles of the image. In this work, we show a proof of concept using images of up to 66-megapixels (8192x8192), saving approximately 50GB of memory per image. Using two public challenge datasets, we demonstrate that CNNs can learn to extract relevant information from these large images and benefit from increasing resolution. We improved the area under the receiver-operating characteristic curve from 0.580 (4MP) to 0.706 (66MP) for metastasis detection in breast cancer (CAMELYON17). We also obtained a Spearman correlation metric approaching state-of-the-art performance on the TUPAC16 dataset, from 0.485 (1MP) to 0.570 (16MP). Code to reproduce a subset of the experiments is available at https://github.com/DIAGNijmegen/StreamingCNN.

arXiv commentIn review

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• A Survey on Deep Learning in Medical Image Analysis
• Improved Regularization of Convolutional Neural Networks with Cutout
• Wide Residual Networks
• Recurrent Models of Visual Attention
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• SmoothGrad: removing noise by adding noise
• Attention-based Deep Multiple Instance Learning
• Training Deep Nets with Sublinear Memory Cost
• Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge
• Neural Image Compression for Gigapixel Histopathology Image Analysis
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Data-dependent Initializations of Convolutional Neural Networks
• Augmenting the Pathology Lab: An Intelligent Whole Slide Image Classification System for the Real World
• Classification and Disease Localization in Histopathology Using Only Global Labels: A Weakly-Supervised Approach
• Processing Megapixel Images with Deep Attention-Sampling Models
• Efficient Memory Management for GPU-based Deep Learning Systems
• Reinforced Auto-Zoom Net: Towards Accurate and Fast Breast Cancer Segmentation in Whole-slide Images
• Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks

Cited by in corpus (10)

• Artificial Intelligence for Digital and Computational Pathology
• DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer
• Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology
• Gigapixel Whole-Slide Images Classification using Locally Supervised Learning
• Extending Unsupervised Neural Image Compression With Supervised Multitask Learning
• Learning from Data Streams: An Overview and Update
• Hard-Attention for Scalable Image Classification
• End-to-end Memory-Efficient Reconstruction for Cone Beam CT
• Applications of the Streaming Networks
• Efficient Classification of Very Large Images with Tiny Objects