1k citations · 3.1k across the 56 of their papers we have counts for
58 papers
Patched Denoising Diffusion Models For High-Resolution Image Synthesis
Zheng Ding, Mengqi Zhang, Jiajun Wu +1
We propose an effective denoising diffusion model for generating high-resolution images (e.g., 1024512), trained on small-size image patches (e.g., 6464). We name o…
Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability
Tyler A. Chang, Zhuowen Tu, Benjamin K. Bergen
How do language models learn to make predictions during pre-training? To study this, we extract learning curves from five autoregressive English language model pre-training runs, f…
DocTr: Document Transformer for Structured Information Extraction in Documents
Haofu Liao, Aruni RoyChowdhury, Weijian Li +6
We present a new formulation for structured information extraction (SIE) from visually rich documents. It aims to address the limitations of existing IOB tagging or graph-based for…
Distilling Large Vision-Language Model with Out-of-Distribution Generalizability
Xuanlin Li, Yunhao Fang, Minghua Liu +3
Large vision-language models have achieved outstanding performance, but their size and computational requirements make their deployment on resource-constrained devices and time-sen…
Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts
Zhaoyang Zhang, Yantao Shen, Kunyu Shi +7
We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks which may interfere with each other, result…
DiffusionRig: Learning Personalized Priors for Facial Appearance Editing
Zheng Ding, Xuaner Zhang, Zhihao Xia +3
We address the problem of learning person-specific facial priors from a small number (e.g., 20) of portrait photos of the same person. This enables us to edit this specific person'…
Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction
Hansheng Chen, Jiatao Gu, Anpei Chen +4
3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images. Despite numerous task-specific methods, developing a compreh…
Point Cloud Recognition with Position-to-Structure Attention Transformers
Zheng Ding, James Hou, Zhuowen Tu
In this paper, we present Position-to-Structure Attention Transformers (PS-Former), a Transformer-based algorithm for 3D point cloud recognition. PS-Former deals with the challenge…
On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning
Yifan Xu, Nicklas Hansen, Zirui Wang +3
Reinforcement Learning (RL) algorithms can solve challenging control problems directly from image observations, but they often require millions of environment interactions to do so…
An In-depth Study of Stochastic Backpropagation
Jun Fang, Mingze Xu, Hao Chen +3
In this paper, we provide an in-depth study of Stochastic Backpropagation (SBP) when training deep neural networks for standard image classification and object detection tasks. Dur…
Semi-supervised Vision Transformers at Scale
Zhaowei Cai, Avinash Ravichandran, Paolo Favaro +5
We study semi-supervised learning (SSL) for vision transformers (ViT), an under-explored topic despite the wide adoption of the ViT architectures to different tasks. To tackle this…
Open-Vocabulary Universal Image Segmentation with MaskCLIP
Zheng Ding, Jieke Wang, Zhuowen Tu
In this paper, we tackle an emerging computer vision task, open-vocabulary universal image segmentation, that aims to perform semantic/instance/panoptic segmentation (background se…
The Geometry of Multilingual Language Model Representations
Tyler A. Chang, Zhuowen Tu, Benjamin K. Bergen
We assess how multilingual language models maintain a shared multilingual representation space while still encoding language-sensitive information in each language. Using XLM-R as…
ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training
Yue Zhao, Yantao Shen, Yuanjun Xiong +5
Negative flips are errors introduced in a classification system when a legacy model is updated. Existing methods to reduce the negative flip rate (NFR) either do so at the expense…
X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks
Zhaowei Cai, Gukyeong Kwon, Avinash Ravichandran +4
In this paper, we study the challenging instance-wise vision-language tasks, where the free-form language is required to align with the objects instead of the whole image. To addre…
Text Spotting Transformers
Xiang Zhang, Yongwen Su, Subarna Tripathi +1
In this paper, we present TExt Spotting TRansformers (TESTR), a generic end-to-end text spotting framework using Transformers for text detection and recognition in the wild. TESTR…
MeMOT: Multi-Object Tracking with Memory
Jiarui Cai, Mingze Xu, Wei Li +4
We propose an online tracking algorithm that performs the object detection and data association under a common framework, capable of linking objects after a long time span. This is…
Contrastive Neighborhood Alignment
Pengkai Zhu, Zhaowei Cai, Yuanjun Xiong +4
We present Contrastive Neighborhood Alignment (CNA), a manifold learning approach to maintain the topology of learned features whereby data points that are mapped to nearby represe…
Towards Panoptic 3D Parsing for Single Image in the Wild
Sainan Liu, Vincent Nguyen, Yuan Gao +2
Performing single image holistic understanding and 3D reconstruction is a central task in computer vision. This paper presents an integrated system that performs dense scene labeli…
ViTGAN: Training GANs with Vision Transformers
Kwonjoon Lee, Huiwen Chang, Lu Jiang +3
Recently, Vision Transformers (ViTs) have shown competitive performance on image recognition while requiring less vision-specific inductive biases. In this paper, we investigate if…
Long Short-Term Transformer for Online Action Detection
Mingze Xu, Yuanjun Xiong, Hao Chen +4
We present Long Short-term TRansformer (LSTR), a temporal modeling algorithm for online action detection, which employs a long- and short-term memory mechanism to model prolonged s…
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models
Tyler A. Chang, Yifan Xu, Weijian Xu +1
In this paper, we detail the relationship between convolutions and self-attention in natural language tasks. We show that relative position embeddings in self-attention layers are…
Compatibility-aware Heterogeneous Visual Search
Rahul Duggal, Hao Zhou, Shuo Yang +4
We tackle the problem of visual search under resource constraints. Existing systems use the same embedding model to compute representations (embeddings) for the query and gallery i…
Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queries
Qi Dong, Zhuowen Tu, Haofu Liao +3
Computer vision applications such as visual relationship detection and human object interaction can be formulated as a composite (structured) set detection problem in which both th…