325 citations · 636 across the 29 of their papers we have counts for
23 papers · 1 filter
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu +4
Instruction tuning unlocks the superior capability of Large Language Models (LLM) to interact with humans. Furthermore, recent instruction-following datasets include images as visu…
AIMS: All-Inclusive Multi-Level Segmentation
Lu Qi, Jason Kuen, Weidong Guo +5
Despite the progress of image segmentation for accurate visual entity segmentation, completing the diverse requirements of image editing applications for different-level region-of-…
MGDoc: Pre-training with Multi-granular Hierarchy for Document Image Understanding
Zilong Wang, Jiuxiang Gu, Chris Tensmeyer +5
Document images are a ubiquitous source of data where the text is organized in a complex hierarchical structure ranging from fine granularity (e.g., words), medium granularity (e.g…
Delving into Out-of-Distribution Detection with Vision-Language Representations
Yifei Ming, Ziyang Cai, Jiuxiang Gu +3
Recognizing out-of-distribution (OOD) samples is critical for machine learning systems deployed in the open world. The vast majority of OOD detection methods are driven by a single…
High-Quality Entity Segmentation
Lu Qi, Jason Kuen, Weidong Guo +5
Dense image segmentation tasks e.g., semantic, panoptic) are useful for image editing, but existing methods can hardly generalize well in an in-the-wild setting where there are unr…
Improving the Reliability for Confidence Estimation
Haoxuan Qu, Yanchao Li, Lin Geng Foo +3
Confidence estimation, a task that aims to evaluate the trustworthiness of the model's prediction output during deployment, has received lots of research attention recently, due to…
Meta Spatio-Temporal Debiasing for Video Scene Graph Generation
Li Xu, Haoxuan Qu, Jason Kuen +2
Video scene graph generation (VidSGG) aims to parse the video content into scene graphs, which involves modeling the spatio-temporal contextual information in the video. However, d…
CA-SSL: Class-Agnostic Semi-Supervised Learning for Detection and Segmentation
Lu Qi, Jason Kuen, Zhe Lin +7
To improve instance-level detection/segmentation performance, existing self-supervised and semi-supervised methods extract either task-unrelated or task-specific training signals f…
LAFITE: Towards Language-Free Training for Text-to-Image Generation
Yufan Zhou, Ruiyi Zhang, Changyou Chen +6
One of the major challenges in training text-to-image generation models is the need of a large number of high-quality image-text pairs. While image samples are often easily accessi…
Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling
Dat Huynh, Jason Kuen, Zhe Lin +2
Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. It is an important step toward reducing laborious human supervision. Most existing…
Multi-Scale Aligned Distillation for Low-Resolution Detection
Lu Qi, Jason Kuen, Jiuxiang Gu +5
In instance-level detection tasks (e.g., object detection), reducing input resolution is an easy option to improve runtime efficiency. However, this option traditionally hurts the…
Open-World Entity Segmentation
Lu Qi, Jason Kuen, Yi Wang +5
We introduce a new image segmentation task, called Entity Segmentation (ES), which aims to segment all visual entities (objects and stuffs) in an image without predicting their sem…
SelfDoc: Self-Supervised Document Representation Learning
Peizhao Li, Jiuxiang Gu, Jason Kuen +5
We propose SelfDoc, a task-agnostic pre-training framework for document image understanding. Because documents are multimodal and are intended for sequential reading, our framework…
Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning
Xiangxi Shi, Xu Yang, Jiuxiang Gu +2
Change Captioning is a task that aims to describe the difference between images with natural language. Most existing methods treat this problem as a difference judgment without the…
Watch It Twice: Video Captioning with a Refocused Video Encoder
Xiangxi Shi, Jianfei Cai, Shafiq Joty +1
With the rapid growth of video data and the increasing demands of various applications such as intelligent video search and assistance toward visually-impaired people, video captio…
Scene Graph Generation with External Knowledge and Image Reconstruction
Jiuxiang Gu, Handong Zhao, Zhe Lin +3
Scene graph generation has received growing attention with the advancements in image understanding tasks such as object detection, attributes and relationship prediction,~\etc. How…
Unpaired Image Captioning via Scene Graph Alignments
Jiuxiang Gu, Shafiq Joty, Jianfei Cai +3
Most of current image captioning models heavily rely on paired image-caption datasets. However, getting large scale image-caption paired data is labor-intensive and time-consuming.…
Video Captioning with Boundary-aware Hierarchical Language Decoding and Joint Video Prediction
Xiangxi Shi, Jianfei Cai, Jiuxiang Gu +1
The explosion of video data on the internet requires effective and efficient technology to generate captions automatically for people who are not able to watch the videos. Despite…
Unpaired Image Captioning by Language Pivoting
Jiuxiang Gu, Shafiq Joty, Jianfei Cai +1
Image captioning is a multimodal task involving computer vision and natural language processing, where the goal is to learn a mapping from the image to its natural language descrip…
Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
Jiuxiang Gu, Jianfei Cai, Shafiq Joty +2
Textual-visual cross-modal retrieval has been a hot research topic in both computer vision and natural language processing communities. Learning appropriate representations for mul…
Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
Jiuxiang Gu, Jianfei Cai, Gang Wang +1
The existing image captioning approaches typically train a one-stage sentence decoder, which is difficult to generate rich fine-grained descriptions. On the other hand, multi-stage…
An Empirical Study of Language CNN for Image Captioning
Jiuxiang Gu, Gang Wang, Jianfei Cai +1
Language Models based on recurrent neural networks have dominated recent image caption generation tasks. In this paper, we introduce a Language CNN model which is suitable for stat…
Recent Advances in Convolutional Neural Networks
Jiuxiang Gu, Zhenhua Wang, Jason Kuen +9
In the last few years, deep learning has led to very good performance on a variety of problems, such as visual recognition, speech recognition and natural language processing. Amon…