Exploration into Translation-Equivariant Image Quantization
arXiv:2112.00384
Abstract
This is an exploratory study that discovers the current image quantization (vector quantization) do not satisfy translation equivariance in the quantized space due to aliasing. Instead of focusing on anti-aliasing, we propose a simple yet effective way to achieve translation-equivariant image quantization by enforcing orthogonality among the codebook embeddings. To explore the advantages of translation-equivariant image quantization, we conduct three proof-of-concept experiments with a carefully controlled dataset: (1) text-to-image generation, where the quantized image indices are the target to predict, (2) image-to-text generation, where the quantized image indices are given as a condition, (3) using a smaller training set to analyze sample efficiency. From the strictly controlled experiments, we empirically verify that the translation-equivariant image quantizer improves not only sample efficiency but also the accuracy over VQGAN up to +11.9% in text-to-image generation and +3.9% in image-to-text generation.
arXiv commentICASSP 2023
References in corpus (12)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Neural Discrete Representation Learning
- Convolutional Sequence to Sequence Learning
- Conditional Image Generation with PixelCNN Decoders
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Pixel Recurrent Neural Networks
- Zero-Shot Text-to-Image Generation
- Why do deep convolutional networks generalize so poorly to small image transformations?
- CPTR: Full Transformer Network for Image Captioning
- Vector-quantized Image Modeling with Improved VQGAN
- Unifying Multimodal Transformer for Bi-directional Image and Text Generation
- X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers