Bi-level Feature Alignment for Versatile Image Translation and Manipulation
arXiv:2107.03021
Abstract
Generative adversarial networks (GANs) have achieved great success in image translation and manipulation. However, high-fidelity image generation with faithful style control remains a grand challenge in computer vision. This paper presents a versatile image translation and manipulation framework that achieves accurate semantic and style guidance in image generation by explicitly building a correspondence. To handle the quadratic complexity incurred by building the dense correspondences, we introduce a bi-level feature alignment strategy that adopts a top- operation to rank block-wise features followed by dense attention between block features which reduces memory cost substantially. As the top- operation involves index swapping which precludes the gradient propagation, we approximate the non-differentiable top- operation with a regularized earth mover's problem so that its gradient can be effectively back-propagated. In addition, we design a novel semantic position encoding mechanism that builds up coordinate for each individual semantic region to preserve texture structures while building correspondences. Further, we design a novel confidence feature injection module which mitigates mismatch problem by fusing features adaptively according to the reliability of built correspondences. Extensive experiments show that our method achieves superior performance qualitatively and quantitatively as compared with the state-of-the-art.
arXiv commentAccepted to ECCV 2022
References in corpus (20)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- Tutorial on Variational Autoencoders
- Pose Guided Person Image Generation
- Universal Correspondence Network
- Image Inpainting via Generative Multi-column Convolutional Neural Networks
- Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distance
- SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objects
- Learning Hierarchical Semantic Image Manipulation through Structured Representations
- Soft-Gated Warping-GAN for Pose-Guided Person Image Synthesis
- Blind Image Super-Resolution via Contrastive Representation Learning
- Scene Text Synthesis for Efficient and Effective Deep Network Training
- Visual Attribute Transfer through Deep Image Analogy
- Towards Realistic 3D Embedding via View Alignment
- Diverse Image Inpainting with Bidirectional and Autoregressive Transformers
- EMLight: Lighting Estimation via Spherical Distribution Approximation
- EDIT: Exemplar-Domain Aware Image-to-Image Translation
- FBC-GAN: Diverse and Flexible Image Synthesis via Foreground-Background Composition
- Semantically Multi-modal Image Synthesis