134 citations · 718 across the 111 of their papers we have counts for
6 papers · 1 filter
DesCo: Learning Object Recognition with Rich Language Descriptions
Liunian Harold Li, Zi-Yi Dou, Nanyun Peng +1
Recent development in vision-language approaches has instigated a paradigm shift in learning visual recognition models from language supervision. These approaches align objects wit…
Masked Path Modeling for Vision-and-Language Navigation
Zi-Yi Dou, Feng Gao, Nanyun Peng
Vision-and-language navigation (VLN) agents are trained to navigate in real-world environments by following natural language instructions. A major challenge in VLN is the limited a…
Generalized Decoding for Pixel, Image, and Language
Xueyan Zou, Zi-Yi Dou, Jianwei Yang +11
We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly. X-Decodert takes as input two types of queries: (i) gen…
Character-Centric Story Visualization via Visual Planning and Token Alignment
Hong Chen, Rujun Han, Te-Lin Wu +2
Story visualization advances the traditional text-to-image generation by enabling multiple image generation based on a complete story. This task requires machines to 1) understand…
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan +9
Vision-language (VL) pre-training has recently received considerable attention. However, most existing end-to-end pre-training approaches either only aim to tackle VL tasks such as…
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Zi-Yi Dou, Yichong Xu, Zhe Gan +9
Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks. While recent work has shown that fully transformer-based VL models can be mo…