STA-VPT: SpatioTemporally Aligned Visual Prompt Tuning

Abstract

Typical methods for visual prompt tuning follow the sequential modeling paradigm originating from NLP, learning a sequence of unordered parameterized tokens as visual prompts, which are then prefixed to the flattened image representation for model adaptation. While such a sequential prompting paradigm has exhibited great promise, it presents two potential limitations. First, the learned visual prompts, presented in an unordered sequential form, are unable to capture the underlying spatial relations in the input image which are crucial for effective image encoding. Second, all prompt tokens serve the same role by performing uniform prompting for all image tokens without distinction, lacking fine-grained prompting capability\---i.e., individualized prompting for different visual tokens to capture region-specific semantic patterns. In this work, we introduce the \mymodel model (\emph{STA-VPT}), a novel visual prompting paradigm, which learns a two-dimensional prompt token map for image prompting or a three-dimensional token volume for video prompting, ensuring spatial (or spatiotemporal) alignment with the input image token map (or video token volume). This alignment enables the visual prompts to preserve the spatial (or spatiotemporal) structure, thereby learning the underlying relations within the visual input. Furthermore, each prompt token serves as a specialized prompting expert and is designated to exclusively prompt for the spatially (or spatiotemporally) corresponding visual tokens. Consequently, our \emph{STA-VPT} is capable of performing individualized prompting for different spatiotemporal regions, potentially improving the prompting performance through fine-grained allocation of prompting capacity, in line with mixture of experts (MoE) principles.

Extension to AAAI 2024, substantially revised version; added new experiments, expanded the methodology section