5 papers
FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models
Haoyang Li, Liang Wang, Siyu Zhou +5
CLIP-based prompt tuning enables pretrained Vision-Language Models (VLMs) to efficiently adapt to downstream tasks. Although existing studies have made significant progress, they p…
Raw Data Matters: Enhancing Prompt Tuning by Internal Augmentation on Vision-Language Models
Haoyang Li, Liang Wang, Chao Wang +4
For CLIP-based prompt tuning, introducing more data as additional knowledge for enhancing fine-tuning process is proved to be an effective approach. Existing data amplification str…
WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents
Siyu Zhou, Tianyi Zhou, Yijun Yang +4
Can we build accurate world models out of large language models (LLMs)? How can world models benefit LLM agents? The gap between the prior knowledge of LLMs and the specified envir…
MAO: Efficient Model-Agnostic Optimization of Prompt Tuning for Vision-Language Models
Haoyang Li, Siyu Zhou, Liang Wang +1
Though CLIP-based prompt tuning significantly enhances pre-trained Vision-Language Models, existing research focuses on reconstructing the model architecture, e.g., additional loss…
WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agents
Siyu Zhou, Tianyi Zhou, Yijun Yang +4
Can large language models (LLMs) directly serve as powerful world models for model-based agents? While the gaps between the prior knowledge of LLMs and the specified environment's…