Debiased Subjective Assessment of Real-World Image Enhancement
arXiv:2106.10080
Abstract
In real-world image enhancement, it is often challenging (if not impossible) to acquire ground-truth data, preventing the adoption of distance metrics for objective quality assessment. As a result, one often resorts to subjective quality assessment, the most straightforward and reliable means of evaluating image enhancement. Conventional subjective testing requires manually pre-selecting a small set of visual examples, which may suffer from three sources of biases: 1) sampling bias due to the extremely sparse distribution of the selected samples in the image space; 2) algorithmic bias due to potential overfitting the selected samples; 3) subjective bias due to further potential cherry-picking test results. This eventually makes the field of real-world image enhancement more of an art than a science. Here we take steps towards debiasing conventional subjective assessment by automatically sampling a set of adaptive and diverse images for subsequent testing. This is achieved by casting sample selection into a joint maximization of the discrepancy between the enhancers and the diversity among the selected input images. Careful visual inspection on the resulting enhanced images provides a debiased ranking of the enhancement algorithms. We demonstrate our subjective assessment method using three popular and practically demanding image enhancement tasks: dehazing, super-resolution, and low-light enhancement.
References in corpus (9)
- DehazeNet: An End-to-End System for Single Image Haze Removal
- DeepXplore: Automated Whitebox Testing of Deep Learning Systems
- ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
- Image Quality Assessment: Unifying Structure and Texture Similarity
- A Bio-Inspired Multi-Exposure Fusion Framework for Low-light Image Enhancement
- Deep Retinex Decomposition for Low-Light Enhancement
- ESRGAN+ : Further Improving Enhanced Super-Resolution Generative Adversarial Network
- Improved Techniques for Learning to Dehaze and Beyond: A Collective Study
- I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively