11 papers
TopoTuner: Topological Finetuning of Large Language Models
Abdulkadir Erol, Yash Mahajan, Vepaul Hariprashad +4
Full fine-tuning remains a strong way to adapt pretrained LLMs, but it updates all weights and can be expensive. LoRA reduces the number of trainable parameters, but it does not di…
Weakly-Supervised Spatiotemporal Anomaly Detection
Urvi Gianchandani, Praveen Tirupattur, Mubarak Shah
In this paper, we explore a weakly supervised method for anomaly detection. Since annotating videos is time-consuming, we only look at weak video-level labels during training. This…
Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference
Bian Sun, Kevin Zhai, Mubarak Shah +1
Diffusion language models (DLMs) have recently emerged as a promising alternative to autoregressive models, primarily due to their ability to enable parallel decoding. Despite this…
VidTAG: Temporally Aligned Video to GPS Geolocalization with Denoising Sequence Prediction at a Global Scale
Parth Parag Kulkarni, Rohit Gupta, Prakash Chandra Chhipa +1
The task of video geolocalization aims to determine the precise GPS coordinates of a video's origin and map its trajectory; with applications in forensics, social media, and explor…
ViLL-E: Video LLM Embeddings for Retrieval
Rohit Gupta, Jayakrishnan Unnikrishnan, Fan Fei +3
Video Large Language Models (VideoLLMs) excel at video understanding tasks where outputs are textual, such as Video Question Answering and Video Captioning. However, they underperf…
Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs
Jinqi Luo, Jinyu Yang, Tal Neiman +5
Multimodal Large Language Models (MLLMs) have been shown to be vulnerable to malicious queries that can elicit unsafe responses. Recent work uses prompt engineering, response class…
Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs
Vishal Narnaware, Animesh Gupta, Kevin Zhai +2
Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucina…
CompLLM: Compression for Long Context Q&A
Gabriele Berton, Jayakrishnan Unnikrishnan, Son Tran +1
Large Language Models (LLMs) face significant computational challenges when processing long contexts due to the quadratic complexity of self-attention. While soft context compressi…
M-LLM Based Video Frame Selection for Efficient Video Understanding
Kai Hu, Feng Gao, Xiaohan Nie +8
Recent advances in Multi-Modal Large Language Models (M-LLMs) show promising results in video reasoning. Popular Multi-Modal Large Language Model (M-LLM) frameworks usually apply n…
CoLLM: A Large Language Model for Composed Image Retrieval
Chuong Huynh, Jinyu Yang, Ashish Tawari +5
Composed Image Retrieval (CIR) is a complex task that aims to retrieve images based on a multimodal query. Typical training data consists of triplets containing a reference image,…
Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
Utsav Singh, Souradip Chakraborty, Wesley A. Suttle +6
Hierarchical reinforcement learning (HRL) enables agents to solve complex, long-horizon tasks by decomposing them into manageable sub-tasks. However, HRL methods face two fundament…