HighRes-net: Recursive Fusion for Multi-Frame Super-Resolution of Satellite Imagery
arXiv:2002.06460
Abstract
Generative deep learning has sparked a new wave of Super-Resolution (SR) algorithms that enhance single images with impressive aesthetic results, albeit with imaginary details. Multi-frame Super-Resolution (MFSR) offers a more grounded approach to the ill-posed problem, by conditioning on multiple low-resolution views. This is important for satellite monitoring of human impact on the planet -- from deforestation, to human rights violations -- that depend on reliable imagery. To this end, we present HighRes-net, the first deep learning approach to MFSR that learns its sub-tasks in an end-to-end fashion: (i) co-registration, (ii) fusion, (iii) up-sampling, and (iv) registration-at-the-loss. Co-registration of low-resolution views is learned implicitly through a reference-frame channel, with no explicit registration mechanism. We learn a global fusion operator that is applied recursively on an arbitrary number of low-resolution pairs. We introduce a registered loss, by learning to align the SR output to a ground-truth through ShiftNet. We show that by learning deep representations of multiple views, we can super-resolve low-resolution signals and enhance Earth Observation data at scale. Our approach recently topped the European Space Agency's MFSR competition on real-world satellite imagery.
arXiv comment15 pages, 5 figures
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Order Matters: Sequence to sequence for sets
- Deep Image Homography Estimation
- Image registration and super resolution from first principles
- Mapping Informal Settlements in Developing Countries with Multi-resolution, Multi-spectral Data
- Learning Disentangled Representations of Satellite Image Time Series
Cited by in corpus (10)
- Multitemporal and multispectral data fusion for super-resolution of Sentinel-2 images
- Multi-temporal speckle reduction with self-supervised deep neural networks
- Predicting Landsat Reflectance with Deep Generative Fusion
- Super-Resolving Beyond Satellite Hardware Using Realistically Degraded Images
- Toward task-driven satellite image super-resolution
- Pix2Streams: Dynamic Hydrology Maps from Satellite-LiDAR Fusion
- Cryo-ZSSR: multiple-image super-resolution based on deep internal learning
- FireSRnet: Geoscience-Driven Super-Resolution of Future Fire Risk from Climate Change
- RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network
- Burst Image Restoration and Enhancement