5 papers
HyperTokens: Controlling Token Dynamics for Continual Video-Language Understanding
Toan Nguyen, Yang Liu, Celso De Melo +1
Continual VideoQA with multimodal LLMs is hindered by interference between tasks and the prohibitive cost of storing task-specific prompts. We introduce HyperTokens, a transformer-…
VAMOS: A Hierarchical Vision-Language-Action Model for Capability-Modulated and Steerable Navigation
Mateo Guaman Castro, Sidharth Rajagopal, Daniel Gorbatov +9
A fundamental challenge in robot navigation lies in learning policies that generalize across diverse environments while conforming to the unique physical constraints and capabiliti…
AHA -- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead
Aiden Chang, Celso De Melo, Stephanie M. Lukin
Real-time understanding of continuous video streams is essential for intelligent agents operating in high-stakes environments, including autonomous vehicles, surveillance drones, a…
Bisecle: Binding and Separation in Continual Learning for Video Language Understanding
Yue Tan, Xiaoqian Hu, Hao Xue +2
Frontier vision-language models (VLMs) have made remarkable improvements in video understanding tasks. However, real-world videos typically exist as continuously evolving data stre…
ViLCo-Bench: VIdeo Language COntinual learning Benchmark
Tianqi Tang, Shohreh Deldari, Hao Xue +2
Video language continual learning involves continuously adapting to information from video and text inputs, enhancing a model's ability to handle new tasks while retaining prior kn…