Deformable Kernel Networks for Joint Image Filtering

Abstract

Joint image filters are used to transfer structural details from a guidance picture used as a prior to a target image, in tasks such as enhancing spatial resolution and suppressing noise. Previous methods based on convolutional neural networks (CNNs) combine nonlinear activations of spatially-invariant kernels to estimate structural details and regress the filtering result. In this paper, we instead learn explicitly sparse and spatially-variant kernels. We propose a CNN architecture and its efficient implementation, called the deformable kernel network (DKN), that outputs sets of neighbors and the corresponding weights adaptively for each pixel. The filtering result is then computed as a weighted average. We also propose a fast version of DKN that runs about seventeen times faster for an image of size 640 x 480. We demonstrate the effectiveness and flexibility of our models on the tasks of depth map upsampling, saliency map upsampling, cross-modality image restoration, texture removal, and semantic segmentation. In particular, we show that the weighted averaging process with sparsely sampled 3 x 3 kernels outperforms the state of the art by a significant margin in all cases.

International Journal of Computer Vision (2020). arXiv admin note: substantial text overlap with arXiv:1903.11286

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Conditional Random Fields as Recurrent Neural Networks
• Multi-Scale Context Aggregation by Dilated Convolutions

Cited by in corpus (5)

• Joint Implicit Image Function for Guided Depth Super-Resolution
• High-resolution Depth Maps Imaging via Attention-based Hierarchical Multi-modal Fusion
• Guided Depth Map Super-resolution: A Survey
• Supersampling of Data from Structured-light Scanner with Deep Learning
• A Guided Upsampling Network for Short Wave Infrared Images Using Graph Regularization