Facial-Sketch Synthesis: A New Challenge

Abstract

This paper aims to conduct a comprehensive study on facial-sketch synthesis (FSS). However, due to the high costs of obtaining hand-drawn sketch datasets, there lacks a complete benchmark for assessing the development of FSS algorithms over the last decade. We first introduce a high-quality dataset for FSS, named FS2K, which consists of 2,104 image-sketch pairs spanning three types of sketch styles, image backgrounds, lighting conditions, skin colors, and facial attributes. FS2K differs from previous FSS datasets in difficulty, diversity, and scalability and should thus facilitate the progress of FSS research. Second, we present the largest-scale FSS investigation by reviewing 89 classical methods, including 25 handcrafted feature-based facial-sketch synthesis approaches, 29 general translation methods, and 35 image-to-sketch approaches. Besides, we elaborate comprehensive experiments on the existing 19 cutting-edge models. Third, we present a simple baseline for FSS, named FSGAN. With only two straightforward components, i.e., facial-aware masking and style-vector expansion, FSGAN surpasses the performance of all previous state-of-the-art models on the proposed FS2K dataset by a large margin. Finally, we conclude with lessons learned over the past years and point out several unsolved challenges. Our code is available at https://github.com/DengPingFan/FSGAN.

arXiv commentAccepted to Machine Intelligence Research (MIR)

References in corpus (28)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Conditional Generative Adversarial Nets
• Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks
• Deep Facial Expression Recognition: A Survey
• Learning Face Representation from Scratch
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)
• MLP-Mixer: An all-MLP Architecture for Vision
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Toward Multimodal Image-to-Image Translation
• A Learned Representation For Artistic Style
• Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• A Neural Representation of Sketch Drawings
• U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation
• Universal Style Transfer via Feature Transforms
• End-to-End Photo-Sketch Generation via Fully Convolutional Representation Learning
• Towards the Automatic Anime Characters Creation with Generative Adversarial Networks
• SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting
• Identity-Aware CycleGAN for Face Photo-Sketch Synthesis and Recognition
• ViTGAN: Training GANs with Vision Transformers
• SofGAN: A Portrait Image Generator with Dynamic Styling
• Comparison and Analysis of Image-to-Image Generative Adversarial Networks: A Survey
• Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation
• PI-REC: Progressive Image Reconstruction Network With Edge and Color Domain
• DeepI2I: Enabling Deep Hierarchical Image-to-Image Translation by Transferring from GANs
• Domain-Specific Mappings for Generative Adversarial Style Transfer

Cited by in corpus (4)

• A survey of synthetic data augmentation methods in computer vision
• Stylized Face Sketch Extraction via Generative Prior with Limited Data
• CLIP4Sketch: Enhancing Sketch to Mugshot Matching through Dataset Augmentation using Diffusion Models
• MixSA: Training-free Reference-based Sketch Extraction via Mixture-of-Self-Attention