Massive Online Crowdsourced Study of Subjective and Objective Picture Quality

Abstract

Most publicly available image quality databases have been created under highly controlled conditions by introducing graded simulated distortions onto high-quality photographs. However, images captured using typical real-world mobile camera devices are usually afflicted by complex mixtures of multiple distortions, which are not necessarily well-modeled by the synthetic distortions found in existing databases. The originators of existing legacy databases usually conducted human psychometric studies to obtain statistically meaningful sets of human opinion scores on images in a stringently controlled visual environment, resulting in small data collections relative to other kinds of image analysis databases. Towards overcoming these limitations, we designed and created a new database that we call the LIVE In the Wild Image Quality Challenge Database, which contains widely diverse authentic image distortions on a large number of images captured using a representative variety of modern mobile devices. We also designed and implemented a new online crowdsourcing system, which we have used to conduct a very large-scale, multi-month image quality assessment subjective study. Our database consists of over 350000 opinion scores on 1162 images evaluated by over 7000 unique human observers. Despite the lack of control over the experimental environments of the numerous study participants, we demonstrate excellent internal consistency of the subjective dataset. We also evaluate several top-performing blind Image Quality Assessment algorithms on it and present insights on how mixtures of distortions challenge both end users as well as automatic perceptual quality prediction models.

16 pages

Cited by in corpus (29)

• Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment
• NIMA: Neural Image Assessment
• Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Network
• KonIQ-10k: Towards an ecologically valid and large-scale IQA database
• KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment
• Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild
• UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content
• Large-Scale Study of Perceptual Video Quality
• Image Quality Assessment using Contrastive Learning
• On the Use of Deep Learning for Blind Image Quality Assessment
• RAPIQUE: Rapid and Accurate Video Quality Prediction of User Generated Content
• Patch-VQ: 'Patching Up' the Video Quality Problem
• GraphIQA: Learning Distortion Graph Representations for Blind Image Quality Assessment
• Blind Quality Assessment for in-the-Wild Images via Hierarchical Feature Fusion and Iterative Mixed Database Training
• Norm-in-Norm Loss with Faster Convergence and Better Performance for Image Quality Assessment
• Study of 3D Virtual Reality Picture Quality
• A Hitchhiker's Guide to Structural Similarity
• Multi-pooled Inception features for no-reference image quality assessment
• Exploiting High-Level Semantics for No-Reference Image Quality Assessment of Realistic Blur Images
• Gap-closing Matters: Perceptual Quality Evaluation and Optimization of Low-Light Image Enhancement
• Deep Ensembling for Perceptual Image Quality Assessment
• Consolidated Dataset and Metrics for High-Dynamic-Range Image Quality
• Bias-Aware Loss for Training Image and Speech Quality Prediction Models from Multiple Datasets
• Helping Visually Impaired People Take Better Quality Pictures
• An Universal Image Attractiveness Ranking Framework
• A Generalised Deep Meta-Learning Model for Automated Quality Control of Cardiovascular Magnetic Resonance Images
• Personalizing image enhancement for critical visual tasks: improved legibility of papyri using color processing and visual illusions
• Annotation-Free Human Sketch Quality Assessment
• Building Reasonable Inference for Vision-Language Models in Blind Image Quality Assessment