HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models

Abstract

Generative models often use human evaluations to measure the perceived quality of their outputs. Automated metrics are noisy indirect proxies, because they rely on heuristics or pretrained embeddings. However, up until now, direct human evaluation strategies have been ad-hoc, neither standardized nor validated. Our work establishes a gold standard human benchmark for generative realism. We construct Human eYe Perceptual Evaluation (HYPE) a human benchmark that is (1) grounded in psychophysics research in perception, (2) reliable across different sets of randomly sampled outputs from a model, (3) able to produce separable model performances, and (4) efficient in cost and time. We introduce two variants: one that measures visual perception under adaptive time constraints to determine the threshold at which a model's outputs appear real (e.g. 250ms), and the other a less expensive variant that measures human error rate on fake and real images sans time constraints. We test HYPE across six state-of-the-art generative adversarial networks and two sampling techniques on conditional and unconditional image generation using four datasets: CelebA, FFHQ, CIFAR-10, and ImageNet. We find that HYPE can track model improvements across training epochs, and we confirm via bootstrap sampling that HYPE rankings are consistent and replicable.

arXiv commenthttps://hype.stanford.edu

Cited by in corpus (16)

• On the Opportunities and Risks of Foundation Models
• Adversarial Text-to-Image Synthesis: A Review
• Classification Accuracy Score for Conditional Generative Models
• Improved Techniques for Training Score-Based Generative Models
• COT-GAN: Generating Sequential Data via Causal Optimal Transport
• On the Evaluation of Conditional GANs
• Adversarial score matching and improved sampling for image generation
• Neural Language Generation: Formulation, Methods, and Evaluation
• Evaluating the Robustness of Collaborative Agents
• Improved Natural Language Generation via Loss Truncation
• ClimateGAN: Raising Climate Change Awareness by Generating Images of Floods
• Conditional Frechet Inception Distance
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Grid Partitioned Attention: Efficient TransformerApproximation with Inductive Bias for High Resolution Detail Generation
• LEGAN: Disentangled Manipulation of Directional Lighting and Facial Expressions by Leveraging Human Perceptual Judgements
• Controlled AutoEncoders to Generate Faces from Voices