172 citations · 199 across the 16 of their papers we have counts for
17 papers
PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation
Kexu Cheng, Zicheng Liu, Mingju Gao +2
Developing physically aware video generation models remains a significant challenge due to the difficulty in capturing diverse physical phenomena, such as thermal dynamics, mechani…
Conditional Text-to-Image Generation with Reference Guidance
Taewook Kim, Ze Wang, Zhengyuan Yang +4
Text-to-image diffusion models have demonstrated tremendous success in synthesizing visually stunning images given textual instructions. Despite remarkable progress in creating hig…
Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization
Zichen Miao, Zhengyuan Yang, Kevin Lin +4
Recent advancements in timestep-distilled diffusion models have enabled high-quality image generation that rivals non-distilled multi-step models, but with significantly fewer infe…
AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition
Minheng Ni, Chenfei Wu, Huaying Yuan +6
With the advancement of generative models, the synthesis of different sensory elements such as music, visuals, and speech has achieved significant realism. However, the approach to…
MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities
Weihao Yu, Zhengyuan Yang, Lingfeng Ren +7
MM-Vet, with open-ended vision-language questions targeting at evaluating integrated capabilities, has become one of the most popular benchmarks for large multimodal model evaluati…
IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generation
Yuanhao Zhai, Kevin Lin, Linjie Li +7
Significant advances have been made in human-centric video generation, yet the joint video-depth generation problem remains underexplored. Most existing monocular depth estimation…
StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis
Zecheng Tang, Chenfei Wu, Zekai Zhang +8
To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model…
A Unified Gaussian Process for Branching and Nested Hyperparameter Optimization
Jiazhao Zhang, Ying Hung, Chung-Ching Lin +1
Choosing appropriate hyperparameters plays a crucial role in the success of neural networks as hyper-parameters directly control the behavior and performance of the training algori…
Bring Metric Functions into Diffusion Models
Jie An, Zhengyuan Yang, Jianfeng Wang +4
We introduce a Cascaded Diffusion Model (Cas-DM) that improves a Denoising Diffusion Probabilistic Model (DDPM) by effectively incorporating additional metric functions in training…
Segment and Caption Anything
Xiaoke Huang, Jianfeng Wang, Yansong Tang +5
We propose a method to efficiently equip the Segment Anything Model (SAM) with the ability to generate regional captions. SAM presents strong generalizability to segment anything w…
MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning
Chaoyi Zhang, Kevin Lin, Zhengyuan Yang +5
We present MM-Narrator, a novel system leveraging GPT-4 with multimodal in-context learning for the generation of audio descriptions (AD). Unlike previous methods that primarily fo…
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
An Yan, Zhengyuan Yang, Wanrong Zhu +9
We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical user interface (GUI) navigation task. MM-Navigator can interact with a smartphone screen as human users,…
OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation
Jie An, Zhengyuan Yang, Linjie Li +5
This work investigates a challenging task named open-domain interleaved image-text generation, which generates interleaved texts and images following an input query. We propose a n…
MM-VID: Advancing Video Understanding with GPT-4V(ision)
Kevin Lin, Faisal Ahmed, Linjie Li +9
We present MM-VID, an integrated system that harnesses the capabilities of GPT-4V, combined with specialized tools in vision, audio, and speech, to facilitate advanced video unders…
The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
Zhengyuan Yang, Linjie Li, Kevin Lin +4
Large multimodal models (LMMs) extend large language models (LLMs) with multi-sensory skills, such as visual understanding, to achieve stronger generic intelligence. In this paper,…
Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generation
Zhengyuan Yang, Jianfeng Wang, Linjie Li +4
We introduce ``Idea to Image,'' a system that enables multimodal iterative self-refinement with GPT-4V(ision) for automatic image design and generation. Humans can quickly identify…
Completing Visual Objects via Bridging Generation and Segmentation
Xiang Li, Yinpeng Chen, Chung-Ching Lin +6
This paper presents a novel approach to object completion, with the primary goal of reconstructing a complete object from its partially visible components. Our method, named MaskCo…