119 citations · 322 across the 25 of their papers we have counts for
25 papers
StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners
Yonglong Tian, Lijie Fan, Phillip Isola +2
We investigate the potential of learning visual representations using synthetic images generated by text-to-image models. This is a natural question in the light of the excellent p…
StyleDrop: Text-to-Image Generation in Any Style
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee +11
Pre-trained large text-to-image models synthesize impressive images with an appropriate use of text prompts. However, ambiguities inherent in natural language and out-of-distributi…
Learning Disentangled Prompts for Compositional Image Synthesis
Kihyuk Sohn, Albert Shaw, Yuan Hao +5
We study domain-adaptive image synthesis, the problem of teaching pretrained image generative models a new style or concept from as few as one image to synthesize novel images, to…
Soundini: Sound-Guided Diffusion for Natural Video Editing
Seung Hyun Lee, Sieun Kim, Innfarn Yoo +6
We propose a method for adding sound-guided visual effects to specific regions of videos with a zero-shot setting. Animating the appearance of the visual effect is challenging beca…
Score-Based Diffusion Models as Principled Priors for Inverse Imaging
Berthy T. Feng, Jamie Smith, Michael Rubinstein +3
Priors are essential for reconstructing images from noisy and/or incomplete measurements. The choice of the prior determines both the quality and uncertainty of recovered images. W…
StraIT: Non-autoregressive Generation with Stratified Image Transformer
Shengju Qian, Huiwen Chang, Yuanzhen Li +3
We propose Stratified Image Transformer(StraIT), a pure non-autoregressive(NAR) generative model that demonstrates superiority in high-quality image synthesis over existing autoreg…
VQ3D: Learning a 3D-Aware Generative Model on ImageNet
Kyle Sargent, Jing Yu Koh, Han Zhang +5
Recent work has shown the possibility of training generative models of 3D content from 2D image collections on small datasets corresponding to a single object class, such as human…
Muse: Text-To-Image Generation via Masked Generative Transformers
Huiwen Chang, Han Zhang, Jarred Barber +9
We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregres…
MAGVIT: Masked Generative Video Transformer
Lijun Yu, Yong Cheng, Kihyuk Sohn +8
We introduce the MAsked Generative VIdeo Transformer, MAGVIT, to tackle various video synthesis tasks with a single model. We introduce a 3D tokenizer to quantize a video into spat…
MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis
Tianhong Li, Huiwen Chang, Shlok Kumar Mishra +3
Generative modeling and representation learning are two key tasks in computer vision. However, these models are typically trained independently, which ignores the potential for eac…
A simple, efficient and scalable contrastive masked autoencoder for learning visual representations
Shlok Mishra, Joshua Robinson, Huiwen Chang +4
We introduce CAN, a simple, efficient and scalable method for self-supervised learning of visual representations. Our framework is a minimal and conceptually clean synthesis of (C)…
Visual Prompt Tuning for Generative Transfer Learning
Kihyuk Sohn, Yuan Hao, José Lezama +5
Transferring knowledge from an image synthesis model trained on a large dataset is a promising direction for learning generative image models from various domains efficiently. Whil…
Improved Masked Image Generation with Token-Critic
José Lezama, Huiwen Chang, Lu Jiang +1
Non-autoregressive generative transformers recently demonstrated impressive image generation performance, and orders of magnitude faster sampling than their autoregressive counterp…
MaskGIT: Masked Generative Image Transformer
Huiwen Chang, Han Zhang, Lu Jiang +2
Generative transformers have experienced rapid popularity growth in the computer vision community in synthesizing high-fidelity and high-resolution images. The best generative tran…
BLT: Bidirectional Layout Transformer for Controllable Layout Generation
Xiang Kong, Lu Jiang, Huiwen Chang +4
Creating visual layouts is a critical step in graphic design. Automatic generation of such layouts is essential for scalable and diverse visual designs. To advance conditional layo…
Pyramid Adversarial Training Improves ViT Performance
Charles Herrmann, Kyle Sargent, Lu Jiang +5
Aggressive data augmentation is a key component of the strong generalization capabilities of Vision Transformer (ViT). One such data augmentation technique is adversarial training…
Palette: Image-to-Image Diffusion Models
Chitwan Saharia, William Chan, Huiwen Chang +5
This paper develops a unified framework for image-to-image translation based on conditional diffusion models and evaluates this framework on four challenging image-to-image transla…
SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting
Varun Jampani, Huiwen Chang, Kyle Sargent +8
Single image 3D photography enables viewers to view a still image from novel viewpoints. Recent approaches combine monocular depth networks with inpainting networks to achieve comp…
ViTGAN: Training GANs with Vision Transformers
Kwonjoon Lee, Huiwen Chang, Lu Jiang +3
Recently, Vision Transformers (ViTs) have shown competitive performance on image recognition while requiring less vision-specific inductive biases. In this paper, we investigate if…
LASR: Learning Articulated Shape Reconstruction from a Monocular Video
Gengshan Yang, Deqing Sun, Varun Jampani +6
Remarkable progress has been made in 3D reconstruction of rigid structures from a video or a collection of images. However, it is still challenging to reconstruct nonrigid structur…
AutoFlow: Learning a Better Training Set for Optical Flow
Deqing Sun, Daniel Vlasic, Charles Herrmann +6
Synthetic datasets play a critical role in pre-training CNN models for optical flow, but they are painstaking to generate and hard to adapt to new applications. To automate the pro…
DVMark: A Deep Multiscale Framework for Video Watermarking
Xiyang Luo, Yinxiao Li, Huiwen Chang +3
Video watermarking embeds a message into a cover video in an imperceptible manner, which can be retrieved even if the video undergoes certain modifications or distortions. Traditio…
Deep 3D-to-2D Watermarking: Embedding Messages in 3D Meshes and Extracting Them from 2D Renderings
Innfarn Yoo, Huiwen Chang, Xiyang Luo +4
Digital watermarking is widely used for copyright protection. Traditional 3D watermarking approaches or commercial software are typically designed to embed messages into 3D meshes,…
Zoom-to-Inpaint: Image Inpainting with High-Frequency Details
Soo Ye Kim, Kfir Aberman, Nori Kanazawa +6
Although deep learning has enabled a huge leap forward in image inpainting, current methods are often unable to synthesize realistic high-frequency details. In this paper, we propo…