Seeing Eye to AI? Applying Deep-Feature-Based Similarity Metrics to Information Visualization

Abstract

Judging the similarity of visualizations is crucial to various applications, such as visualization-based search and visualization recommendation systems. Recent studies show deep-feature-based similarity metrics correlate well with perceptual judgments of image similarity and serve as effective loss functions for tasks like image super-resolution and style transfer. We explore the application of such metrics to judgments of visualization similarity. We extend a similarity metric using five ML architectures and three pre-trained weight sets. We replicate results from previous crowd-sourced studies on scatterplot and visual channel similarity perception. Notably, our metric using pre-trained ImageNet weights outperformed gradient-descent tuned MS-SSIM, a multi-scale similarity metric based on luminance, contrast, and structure. Our work contributes to understanding how deep-feature-based metrics can enhance similarity assessments in visualization, potentially improving visual analysis tools and techniques. Supplementary materials are available at https://osf.io/dj2ms.

arXiv commentAccepted at CHI 2025

References in corpus (12)

• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Image Quality Assessment: Unifying Structure and Texture Similarity
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• A Survey on ML4VIS: Applying Machine Learning Advances to Data Visualization
• Getting aligned on representational alignment
• Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization
• Do better ImageNet classifiers assess perceptual similarity better?
• Transforming Neural Network Visual Representations to Predict Human Judgments of Similarity
• An Evaluation-Focused Framework for Visualization Recommendation Algorithms
• A Systematic Performance Analysis of Deep Perceptual Loss Networks: Breaking Transfer Learning Conventions
• PieAPP: Perceptual Image-Error Assessment through Pairwise Preference
• Enriching ImageNet with Human Similarity Judgments and Psychological Embeddings