VTAMIQ: Transformers for Attention Modulated Image Quality Assessment
arXiv:2110.01655
Abstract
Following the major successes of self-attention and Transformers for image analysis, we investigate the use of such attention mechanisms in the context of Image Quality Assessment (IQA) and propose a novel full-reference IQA method, Vision Transformer for Attention Modulated Image Quality (VTAMIQ). Our method achieves competitive or state-of-the-art performance on the existing IQA datasets and significantly outperforms previous metrics in cross-database evaluations. Most patch-wise IQA methods treat each patch independently; this partially discards global information and limits the ability to model long-distance interactions. We avoid this problem altogether by employing a transformer to encode a sequence of patches as a single global representation, which by design considers interdependencies between patches. We rely on various attention mechanisms -- first with self-attention within the Transformer, and second with channel attention within our difference modulation network -- specifically to reveal and enhance the more salient features throughout our architecture. With large-scale pre-training for both classification and IQA tasks, VTAMIQ generalizes well to unseen sets of images and distortions, further demonstrating the strength of transformer-based networks for vision modelling.
arXiv comment11 pages, 6 figures
References in corpus (10)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Deep Residual Learning for Image Recognition
- Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index
- A Haar Wavelet-Based Perceptual Similarity Index for Image Quality Assessment
- Stand-Alone Self-Attention in Vision Models
- Long Short-Term Memory-Networks for Machine Reading
- Mean Deviation Similarity Index: Efficient and Reliable Full-Reference Image Quality Evaluator
- Hallucinated-IQA: No-Reference Image Quality Assessment via Adversarial Learning
- DeepFL-IQA: Weak Supervision for Deep IQA Feature Learning