Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration

Abstract

Image restoration is a long-standing problem in low-level computer vision with many interesting applications. We describe a flexible learning framework based on the concept of nonlinear reaction diffusion models for various image restoration problems. By embodying recent improvements in nonlinear diffusion models, we propose a dynamic nonlinear reaction diffusion model with time-dependent parameters (\ie, linear filters and influence functions). In contrast to previous nonlinear diffusion models, all the parameters, including the filters and the influence functions, are simultaneously learned from training data through a loss based approach. We call this approach TNRD -- \textit{Trainable Nonlinear Reaction Diffusion}. The TNRD approach is applicable for a variety of image restoration tasks by incorporating appropriate reaction force. We demonstrate its capabilities with three representative applications, Gaussian image denoising, single image super resolution and JPEG deblocking. Experiments show that our trained nonlinear diffusion models largely benefit from the training of the parameters and finally lead to the best reported performance on common test datasets for the tested applications. Our trained models preserve the structural simplicity of diffusion models and take only a small number of diffusion steps, thus are highly efficient. Moreover, they are also well-suited for parallel computation on GPUs, which makes the inference procedure extremely fast.

14 pages, 13 figures, to appear in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

References in corpus (3)

• Learning Activation Functions to Improve Deep Neural Networks
• Insights into analysis operator learning: From patch-based sparse models to higher-order MRFs
• Revisiting loss-specific training of filter-based MRFs for image restoration

Cited by in corpus (217)

• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising
• A Review of Convolutional Neural Networks for Inverse Problems in Imaging
• Denoising Prior Driven Deep Neural Network for Image Restoration
• DeepISP: Towards Learning an End-to-End Image Processing Pipeline
• Non-Local Recurrent Network for Image Restoration
• Plug and play methods for magnetic resonance imaging (long version)
• Practical Blind Image Denoising via Swin-Conv-UNet and Data Synthesis
• Enhancing Quality for HEVC Compressed Videos
• An Online Plug-and-Play Algorithm for Regularized Image Reconstruction
• Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
• MemNet: A Persistent Memory Network for Image Restoration
• Noisy-As-Clean: Learning Self-supervised Denoising from the Corrupted Image
• Enhanced CNN for image denoising
• Deep Graph-Convolutional Image Denoising
• Deep Learning-Based Video Coding: A Review and A Case Study
• WINNet: Wavelet-inspired Invertible Network for Image Denoising
• Blind Universal Bayesian Image Denoising with Gaussian Noise Level Learning
• Learning Deep CNN Denoiser Prior for Image Restoration
• DVDnet: A Fast Network for Deep Video Denoising
• Pre-Trained Image Processing Transformer
• Dynamic Path-Controllable Deep Unfolding Network for Compressive Sensing
• Nonlocality-Reinforced Convolutional Neural Networks for Image Denoising
• Can learning from natural image denoising be used for seismic data interpolation?
• Path-Restore: Learning Network Path Selection for Image Restoration
• Memory-Augmented Deep Unfolding Network for Compressive Sensing
• Image Compressed Sensing Using Non-local Neural Network
• Deep BCD-Net Using Identical Encoding-Decoding CNN Structures for Iterative Image Recovery
• A Comprehensive Benchmark for Single Image Compression Artifacts Reduction
• COLA-Net: Collaborative Attention Network for Image Restoration
• NLH: A Blind Pixel-level Non-local Method for Real-world Image Denoising
• Learning to Optimize: A Primer and A Benchmark
• Kindling the Darkness: A Practical Low-light Image Enhancer
• Deep Learning Methods for Solving Linear Inverse Problems: Research Directions and Paradigms
• SwinIR: Image Restoration Using Swin Transformer
• Learning Converged Propagations with Deep Prior Ensemble for Image Enhancement
• Optical Fringe Patterns Filtering Based on Multi-Stage Convolution Neural Network
• Class-Aware Fully-Convolutional Gaussian and Poisson Denoising
• Pyramid Attention Networks for Image Restoration
• Variational Deep Image Restoration
• Deep Mean-Shift Priors for Image Restoration
• Toward Convolutional Blind Denoising of Real Photographs
• Residual Dense Network for Image Restoration
• Accurate and Lightweight Image Super-Resolution with Model-Guided Deep Unfolding Network
• Convolutional Analysis Operator Learning: Acceleration and Convergence
• Non-Convex Weighted Lp Minimization based Group Sparse Representation Framework for Image Denoising
• Fixed-Point and Objective Convergence of Plug-and-Play Algorithms
• Attribute Artifacts Removal for Geometry-based Point Cloud Compression
• Learning to See in the Dark
• VPNet: Variable Projection Networks
• Model-Based Image Signal Processors via Learnable Dictionaries
• Noise2Same: Optimizing A Self-Supervised Bound for Image Denoising
• Regularized Fourier Ptychography using an Online Plug-and-Play Algorithm
• DCT2net: an interpretable shallow CNN for image denoising
• Deep Convolutional Denoising of Low-Light Images
• Multi-level Wavelet-CNN for Image Restoration
• Hyperspectral Image Denoising Using Non-convex Local Low-rank and Sparse Separation with Spatial-Spectral Total Variation Regularization
• Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE
• Robust and interpretable blind image denoising via bias-free convolutional neural networks
• A Novel Image Denoising Algorithm Using Concepts of Quantum Many-Body Theory
• Super-Resolution of License Plate Images Using Attention Modules and Sub-Pixel Convolution Layers
• A deep convolutional neural network for salt-and-pepper noise removal using selective convolutional blocks
• Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior
• Investigating Task-driven Latent Feasibility for Nonconvex Image Modeling
• Non-Local Video Denoising by CNN
• Learning Filter Bank Sparsifying Transforms
• Deep Learning on Image Denoising: An overview
• Plug-and-Play Quantum Adaptive Denoiser for Deconvolving Poisson Noisy Images
• Block-Matching Convolutional Neural Network for Image Denoising
• When Image Denoising Meets High-Level Vision Tasks: A Deep Learning Approach
• Ghost Imaging Based on Recurrent Neural Network
• Stable Deep MRI Reconstruction using Generative Priors
• NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results
• Rotation Equivariant Proximal Operator for Deep Unfolding Methods in Image Restoration
• DnSwin: Toward Real-World Denoising via Continuous Wavelet Sliding-Transformer
• Attention Cube Network for Image Restoration
• Combining Attention Module and Pixel Shuffle for License Plate Super-Resolution
• Transfer Learning from Synthetic to Real-Noise Denoising with Adaptive Instance Normalization
• Connecting Image Denoising and High-Level Vision Tasks via Deep Learning
• Uncovering the Over-smoothing Challenge in Image Super-Resolution: Entropy-based Quantification and Contrastive Optimization
• Neumann Networks for Inverse Problems in Imaging
• Deep artifact learning for compressed sensing and parallel MRI
• A New Recurrent Plug-and-Play Prior Based on the Multiple Self-Similarity Network
• Learning Convex Regularizers for Optimal Bayesian Denoising
• Toward DNN of LUTs: Learning Efficient Image Restoration with Multiple Look-Up Tables
• On Maximum-a-Posteriori estimation with Plug & Play priors and stochastic gradient descent
• NBNet: Noise Basis Learning for Image Denoising with Subspace Projection
• Momentum-Net: Fast and convergent iterative neural network for inverse problems
• Image Denoising using Attention-Residual Convolutional Neural Networks
• Deep Algorithm Unrolling for Blind Image Deblurring
• Dual Adversarial Network: Toward Real-world Noise Removal and Noise Generation
• Chaining Identity Mapping Modules for Image Denoising
• Distributed Learning and Inference with Compressed Images
• Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser
• Deep Super-Resolution Network for Single Image Super-Resolution with Realistic Degradations
• Nest-DGIL: Nesterov-optimized Deep Geometric Incremental Learning for CS Image Reconstruction
• Multi-band Weighted $l_p$ Norm Minimization for Image Denoising
• Total Deep Variation: A Stable Regularizer for Inverse Problems
• Unsupervised Image Noise Modeling with Self-Consistent GAN
• A Multi-Head Convolutional Neural Network With Multi-path Attention improves Image Denoising
• Invertible Denoising Network: A Light Solution for Real Noise Removal
• DMCNN: Dual-Domain Multi-Scale Convolutional Neural Network for Compression Artifacts Removal
• Rank-One Network: An Effective Framework for Image Restoration
• Proximal Alternating Direction Network: A Globally Converged Deep Unrolling Framework
• Semantic Image Completion and Enhancement using Deep Learning
• Adaptive Denoising via GainTuning
• A Comprehensive Comparison of Multi-Dimensional Image Denoising Methods
• Designing and Training of A Dual CNN for Image Denoising
• Deep Learning Techniques for Inverse Problems in Imaging
• Progressive Training of Multi-level Wavelet Residual Networks for Image Denoising
• NormalNet: Learning-based Normal Filtering for Mesh Denoising
• Group Sparsity Residual Constraint for Image Denoising
• External Patch-Based Image Restoration Using Importance Sampling
• An ${\ell }^1$-Plug-and-Play Approach for MPI Using a Zero Shot Denoiser with Evaluation on the 3D Open MPI Dataset
• Simultaneous Fidelity and Regularization Learning for Image Restoration
• Linear Combinations of Patches are Unreasonably Effective for Single-Image Denoising
• A plug-and-play framework for curvilinear structure segmentation based on a learned reconnecting regularization
• Learning Deep Gradient Descent Optimization for Image Deconvolution
• LEARN: Learned Experts' Assessment-based Reconstruction Network for Sparse-data CT
• Deep Class Aware Denoising
• CNN-Based Real-Time Parameter Tuning for Optimizing Denoising Filter Performance
• DPW-SDNet: Dual Pixel-Wavelet Domain Deep CNNs for Soft Decoding of JPEG-Compressed Images
• A General Decoupled Learning Framework for Parameterized Image Operators
• A Set-Theoretic Study of the Relationships of Image Models and Priors for Restoration Problems
• PixelRL: Fully Convolutional Network with Reinforcement Learning for Image Processing
• Self-supervised Hyperspectral Image Restoration using Separable Image Prior
• Multiple Description Convolutional Neural Networks for Image Compression
• Solving Inverse Problems by Joint Posterior Maximization with a VAE Prior
• Geometric thermodynamics of reaction-diffusion systems: Thermodynamic trade-off relations and optimal transport for pattern formation
• Spatial-Adaptive Network for Single Image Denoising
• Probabilistic Residual Learning for Aleatoric Uncertainty in Image Restoration
• Texture Generation with Neural Cellular Automata
• Accelerating Neutron Scattering Data Collection and Experiments Using AI Deep Super-Resolution Learning
• Learning Raw Image Denoising with Bayer Pattern Unification and Bayer Preserving Augmentation
• Modeling Dual-Exposure Quad-Bayer Patterns for Joint Denoising and Deblurring
• CFSNet: Toward a Controllable Feature Space for Image Restoration
• The Power of Complementary Regularizers: Image Recovery via Transform Learning and Low-Rank Modeling
• End-to-end Interpretable Learning of Non-blind Image Deblurring
• Enhancing super-resolution ultrasound localisation through multi-frame deconvolution exploiting spatiotemporal coherence
• Translating Diffusion, Wavelets, and Regularisation into Residual Networks
• An Optimal Control Approach to Early Stopping Variational Methods for Image Restoration
• Solving RED with Weighted Proximal Methods
• Identifying Recurring Patterns with Deep Neural Networks for Natural Image Denoising
• Restoring Images with Unknown Degradation Factors by Recurrent Use of a Multi-branch Network
• A Fast Stochastic Plug-and-Play ADMM for Imaging Inverse Problems
• FINO: Flow-based Joint Image and Noise Model
• DIVA: Deep Unfolded Network from Quantum Interactive Patches for Image Restoration
• Feasibility-based Fixed Point Networks
• Beyond Deep Residual Learning for Image Restoration: Persistent Homology-Guided Manifold Simplification
• Deep Unfolding of a Proximal Interior Point Method for Image Restoration
• Multi-level Wavelet Convolutional Neural Networks
• LSD${}_2$ -- Joint Denoising and Deblurring of Short and Long Exposure Images with CNNs
• Evolutionary Variational Optimization of Generative Models
• On the Convergence of Learning-based Iterative Methods for Nonconvex Inverse Problems
• New Computational Techniques for a Faster Variation of BM3D Image Denoising
• Phase Retrieval using Expectation Consistent Signal Recovery Algorithm based on Hypernetwork
• Practical Deep Raw Image Denoising on Mobile Devices
• Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures
• Reconstructing the Noise Manifold for Image Denoising
• Fully Trainable and Interpretable Non-Local Sparse Models for Image Restoration
• Deep Decomposition Learning for Inverse Imaging Problems
• Image Restoration Using Deep Regulated Convolutional Networks
• R3L: Connecting Deep Reinforcement Learning to Recurrent Neural Networks for Image Denoising via Residual Recovery
• Meta-Optimization of Deep CNN for Image Denoising Using LSTM
• Sub-pixel face landmarks using heatmaps and a bag of tricks
• Structured and Localized Image Restoration
• Concatenated Attention Neural Network for Image Restoration
• End-to-End JPEG Decoding and Artifacts Suppression Using Heterogeneous Residual Convolutional Neural Network
• Lightweight image super-resolution with enhanced CNN
• Convolutional Neural Networks Analyzed via Inverse Problem Theory and Sparse Representations
• Regularization by Denoising: Clarifications and New Interpretations
• A Multiscale Image Denoising Algorithm Based On Dilated Residual Convolution Network
• A lightweight convolutional neural network for image denoising with fine details preservation capability
• Toward Interactive Modulation for Photo-Realistic Image Restoration
• On-Demand Learning for Deep Image Restoration
• Improved low-count quantitative PET reconstruction with an iterative neural network
• An Effective Image Restorer: Denoising and Luminance Adjustment for Low-photon-count Imaging
• Combinatorial Optimization for Panoptic Segmentation: A Fully Differentiable Approach
• Network Architecture Search for Face Enhancement
• Convolutional versus Self-Organized Operational Neural Networks for Real-World Blind Image Denoising
• An Optimization Framework with Flexible Inexact Inner Iterations for Nonconvex and Nonsmooth Programming
• Distributed Deep Learning for Medical Image Denoising with Data Obfuscation
• KRNET: Image Denoising with Kernel Regulation Network
• Learning Model-Blind Temporal Denoisers without Ground Truths
• ${\ell }_1$DecNet+: A new architecture framework by ${\ell }_1$ decomposition and iteration unfolding for sparse feature segmentation
• A Statistical Framework to Investigate the Optimality of Signal-Reconstruction Methods
• Scale-wise Convolution for Image Restoration
• Soft Delivery: Survey on A New Paradigm for Wireless and Mobile Multimedia Streaming
• Mixed-Resolution Image Representation and Compression with Convolutional Neural Networks
• Super-resolving Compressed Images via Parallel and Series Integration of Artifact Reduction and Resolution Enhancement
• NFCNN: Toward a Noise Fusion Convolutional Neural Network for Image Denoising
• Towards Boosting the Channel Attention in Real Image Denoising : Sub-band Pyramid Attention
• Bayesian Imaging With Data-Driven Priors Encoded by Neural Networks: Theory, Methods, and Algorithms
• Learning Deep Interleaved Networks with Asymmetric Co-Attention for Image Restoration
• Identity Enhanced Residual Image Denoising
• A Research and Strategy of Remote Sensing Image Denoising Algorithms
• Lattice Fusion Networks for Image Denoising
• Searching Efficient Model-guided Deep Network for Image Denoising
• Dense-Sparse Deep Convolutional Neural Networks Training for Image Denoising
• SSCAN: A Spatial-spectral Cross Attention Network for Hyperspectral Image Denoising
• Learning to Jointly Deblur, Demosaick and Denoise Raw Images
• Improved PDE Models for Image Restoration through Backpropagation
• Asymmetric CNN for image super-resolution
• Robust lEarned Shrinkage-Thresholding (REST): Robust unrolling for sparse recover
• Details Preserving Deep Collaborative Filtering-Based Method for Image Denoising
• Boosting the Performance of Plug-and-Play Priors via Denoiser Scaling
• Enhancing and Learning Denoiser without Clean Reference
• Disentangling Noise from Images: A Flow-Based Image Denoising Neural Network
• A Critical Analysis of Patch Similarity Based Image Denoising Algorithms
• Product-of-Gaussian-Mixture Diffusion Models for Joint Nonlinear MRI Reconstruction
• Multidimensional TV-Stokes for image processing
• Variational Deep Image Denoising
• Generalized Intersection Algorithms with Fixpoints for Image Decomposition Learning
• Learning Deep Image Priors for Blind Image Denoising
• Cross-Scale Residual Network for Multiple Tasks:Image Super-resolution, Denoising, and Deblocking
• Acceleration of RED via Vector Extrapolation
• Joint multiband deconvolution for Euclid and Vera C. Rubin images