244 citations · 389 across the 28 of their papers we have counts for
28 papers
SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy
Peiyan Li, Yixiang Chen, Yuan Xu +13
Robotic manipulation requires understanding both the 3D spatial structure of the environment and its temporal evolution, yet most existing policies neglect one or both aspects. The…
How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing
Huanyu Zhang, Xuehai Bai, Chengzu Li +9
Recent generative models have achieved remarkable progress in image editing. However, existing systems and benchmarks remain largely text-guided. In contrast, human communication i…
SimScale: Learning to Drive via Real-World Simulation at Scale
Haochen Tian, Tianyu Li, Haochen Liu +11
Achieving fully autonomous driving systems requires learning rational decisions in a wide span of scenarios, including safety-critical and out-of-distribution ones. However, such c…
Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs
Huanyu Zhang, Wenshan Wu, Chengzu Li +9
While Multimodal Large Language Models (MLLMs) excel at visual understanding, they often struggle in complex scenarios that require visual planning and imagination. Inspired by how…
MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models
Wulin Xie, Yi-Fan Zhang, Chaoyou Fu +6
Existing MLLM benchmarks face significant challenges in evaluating Unified MLLMs (U-MLLMs) due to: 1) lack of standardized benchmarks for traditional tasks, leading to inconsistent…
Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
Huanyu Zhang, Chengzu Li, Wenshan Wu +8
Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in general vision-language tasks. However, recent studies have exposed critical limitations in the…
Aligning Multimodal LLM with Human Preference: A Survey
Tao Yu, Yi-Fan Zhang, Chaoyou Fu +14
Large language models (LLMs) can handle a wide variety of general tasks with simple prompts, without the need for task-specific training. Multimodal Large Language Models (MLLMs),…
MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
Yi-Fan Zhang, Tao Yu, Haochen Tian +17
Despite notable advancements in Multimodal Large Language Models (MLLMs), most state-of-the-art models have not undergone thorough alignment with human preferences. This gap exists…
TimeRAF: Retrieval-Augmented Foundation model for Zero-shot Time Series Forecasting
Huanyu Zhang, Chang Xu, Yi-Fan Zhang +4
Time series forecasting plays a crucial role in data mining, driving rapid advancements across numerous industries. With the emergence of large models, time series foundation model…
MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian +10
Comprehensive evaluation of Multimodal Large Language Models (MLLMs) has recently garnered widespread attention in the research community. However, we observe that existing benchma…
VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark
Han Huang, Haitian Zhong, Tao Yu +4
Recently, knowledge editing on large language models (LLMs) has received considerable attention. Compared to this, editing Large Vision-Language Models (LVLMs) faces extra challeng…
Debiasing Multimodal Large Language Models via Penalization of Language Priors
YiFan Zhang, Yang Shi, Weichen Yu +6
In the realms of computer vision and natural language processing, Multimodal Large Language Models (MLLMs) have become indispensable tools, proficient in generating textual respons…
Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models
Junfei Wu, Qiang Liu, Ding Wang +4
Object hallucination has been an Achilles' heel which hinders the broader applications of large vision-language models (LVLMs). Object hallucination refers to the phenomenon that t…
Assaying on the Robustness of Zero-Shot Machine-Generated Text Detectors
Yi-Fan Zhang, Zhang Zhang, Liang Wang +2
To combat the potential misuse of Natural Language Generation (NLG) technology, a variety of algorithms have been developed for the detection of AI-generated texts. Traditionally,…
AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis
Dongze Li, Kang Zhao, Wei Wang +4
Audio-driven talking head synthesis is a promising topic with wide applications in digital human, film making and virtual reality. Recent NeRF-based approaches have shown superiori…
OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling
Yi-Fan Zhang, Qingsong Wen, Xue Wang +6
Online updating of time series forecasting models aims to address the concept drifting problem by efficiently updating forecasting models based on streaming data. Many algorithms a…
End-to-end Alternating Optimization for Real-World Blind Super Resolution
Zhengxiong Luo, Yan Huang, Shang Li +2
Blind Super-Resolution (SR) usually involves two sub-problems: 1) estimating the degradation of the given low-resolution (LR) image; 2) super-resolving the LR image to its high-res…
GaFET: Learning Geometry-aware Facial Expression Translation from In-The-Wild Images
Tianxiang Ma, Bingchuan Li, Qian He +2
While current face animation methods can manipulate expressions individually, they suffer from several limitations. The expressions manipulated by some motion-based facial reenactm…
Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization
Jian Liang, Lijun Sheng, Zhengbo Wang +2
The emergence of vision-language models, such as CLIP, has spurred a significant research effort towards their application for downstream supervised learning tasks. Although some p…
Improving Zero-Shot Generalization for CLIP with Synthesized Prompts
Zhengbo Wang, Jian Liang, Ran He +3
With the growing interest in pretrained vision-language models like CLIP, recent research has focused on adapting these models to downstream tasks. Despite achieving promising resu…
AdaNPC: Exploring Non-Parametric Classifier for Test-Time Adaptation
Yi-Fan Zhang, Xue Wang, Kexin Jin +5
Many recent machine learning tasks focus to develop models that can generalize to unseen distributions. Domain generalization (DG) has become one of the key topics in various field…
Semantic Prompt for Few-Shot Image Recognition
Wentao Chen, Chenyang Si, Zhang Zhang +3
Few-shot learning is a challenging problem since only a few examples are provided to recognize a new class. Several recent studies exploit additional semantic information, e.g. tex…
Exploiting Semantic Attributes for Transductive Zero-Shot Learning
Zhengbo Wang, Jian Liang, Zilei Wang +1
Zero-shot learning (ZSL) aims to recognize unseen classes by generalizing the relation between visual features and semantic attributes learned from the seen classes. A recent parad…
A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts
Jian Liang, Ran He, Tieniu Tan
Machine learning methods strive to acquire a robust model during the training process that can effectively generalize to test samples, even in the presence of distribution shifts.…