How Can Objects Help Video-Language Understanding?

Zitian Tang, Shijie Wang, Junho Cho +2

Do we still need to represent objects explicitly in multimodal large language models (MLLMs)? To one extreme, pre-trained encoders convert images into visual tokens, with which obj…