Combining Progressive Rethinking and Collaborative Learning: A Deep Framework for In-Loop Filtering

Abstract

In this paper, we aim to address issues of (1) joint spatial-temporal modeling and (2) side information injection for deep-learning based in-loop filter. For (1), we design a deep network with both progressive rethinking and collaborative learning mechanisms to improve quality of the reconstructed intra-frames and inter-frames, respectively. For intra coding, a Progressive Rethinking Network (PRN) is designed to simulate the human decision mechanism for effective spatial modeling. Our designed block introduces an additional inter-block connection to bypass a high-dimensional informative feature before the bottleneck module across blocks to review the complete past memorized experiences and rethinks progressively. For inter coding, the current reconstructed frame interacts with reference frames (peak quality frame and the nearest adjacent frame) collaboratively at the feature level. For (2), we extract both intra-frame and inter-frame side information for better context modeling. A coarse-to-fine partition map based on HEVC partition trees is built as the intra-frame side information. Furthermore, the warped features of the reference frames are offered as the inter-frame side information. Our PRN with intra-frame side information provides 9.0% BD-rate reduction on average compared to HEVC baseline under All-intra (AI) configuration. While under Low-Delay B (LDB), Low-Delay P (LDP) and Random Access (RA) configuration, our PRN with inter-frame side information provides 9.0%, 10.6% and 8.0% BD-rate reduction on average respectively. Our project webpage is https://dezhao-wang.github.io/PRN-v2/.

arXiv commentAccepted for publication in IEEE Transactions on Image Processing (TIP). Website available at https://dezhao-wang.github.io/PRN-v2/

References in corpus (8)

• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising
• Video Enhancement with Task-Oriented Flow
• A Convolutional Neural Network Approach for Post-Processing in HEVC Intra Coding
• CAS-CNN: A Deep Convolutional Neural Network for Image Compression Artifact Suppression
• A DenseNet Based Approach for Multi-Frame In-Loop Filter in HEVC
• Enhancing HEVC Compressed Videos with a Partition-masked Convolutional Neural Network
• Spatial-Temporal Residue Network Based In-Loop Filter for Video Coding