1.7k citations · 2.2k across the 42 of their papers we have counts for
48 papers
Optimal Hydra: Model Heterogeneous Federated Learning for LLMs using Data-Informed Hydra-Nets
Gwen Legate, Rahaf Aljundi, Eugene Belilovsky
Large language models (LLMs) trained on vast amounts of data have demonstrated impressive performance over a range of tasks, but future performance gains will require increasing am…
DemoBridge: A Simulation-in-the-Loop Toolkit for Single-View Human Demonstration Retargeting
Zehao Wang, Fabien Despinoy, Sergey Zakharov +2
We present DemoBridge, an toolkit that turns a single-view RGB stereo recording of a human hand demonstration into an executable, physics-validated robot-arm trajectory. Retargetin…
Continual Self-Improvement with Lightweight Experiential Latent Memories
Vaggelis Dorovatas, Nancy Kalaj, Rahaf Aljundi
Large language models achieve strong reasoning performance by scaling inference-time compute, yet remain fundamentally stateless, discarding the rich, self-produced reasoning trace…
Thinking Past the Answer: Evaluating Harmful Overthinking in Large Reasoning Models
Simone Caldarella, Davide Talon, Rahaf Aljundi +2
Large Reasoning Models (LRMs) improve performance by generating explicit intermediate reasoning traces through increased test-time compute, yet the assumption that longer reasoning…
From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models
Christian Gumbsch, Leonardo Barcellona, Lennard Schünemann +7
Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics. Recent work has explored…
Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observations
Andrii Zadaianchuk, Leonardo Barcellona, Lennard Schuenemann +7
Accurately reconstructing complex full multi-object scenes from sparse observations remains a core challenge in computer vision and a key step toward scalable and reliable simulati…
SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models
Quentin Guimard, Federico Bartsch, Simone Caldarella +3
Models that bridge vision and language, such as CLIP, are key components of multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biase…
Ego: Embedding-Guided Personalization of Vision-Language Models
Soroush Seifi, Simon Gardier, Vaggelis Dorovatas +2
AI assistants that support humans in daily life are becoming increasingly feasible, driven by the rapid advancements in multimodal language models. A key challenge lies in overcomi…
Position: Modular Memory is the Key to Continual Learning Agents
Vaggelis Dorovatas, Malte Schwerin, Andrew D. Bagdanov +21
Foundation models have transformed machine learning through large-scale pretraining and increased test-time compute. Despite surpassing human performance in several domains, these…
OMNI-LEAK: Orchestrator Multi-Agent Network Induced Data Leakage
Akshat Naik, Jay Culligan, Yarin Gal +4
As Large Language Model (LLM) agents become more capable, their coordinated use in the form of multi-agent systems is anticipated to emerge as a practical paradigm. Prior work has…
Evaluating the Impact of Post-Training Quantization on Reliable VQA with Multimodal LLMs
Paul Jonas Kurz, Tobias Jan Wieczorek, Mohamed A. Abdelsalam +2
Multimodal Large Language Models (MLLM) are increasingly deployed in domains where both reliability and efficiency are critical. However, current models remain overconfident, produ…
Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning
Gunshi Gupta, Karmesh Yadav, Zsolt Kira +2
To enable embodied agents to operate effectively over extended timeframes, it is crucial to develop models that form and access memories to stay contextualized in their environment…
When Data Falls Short: Grokking Below the Critical Threshold
Vaibhav Singh, Eugene Belilovsky, Rahaf Aljundi
In this paper, we investigate the phenomenon of grokking, where models exhibit delayed generalization following overfitting on training data. We focus on data-scarce regimes where…
Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs
Vaggelis Dorovatas, Soroush Seifi, Gunshi Gupta +1
Video Large Language Models (Video-LLMs) excel at understanding videos in-context, provided they have full access to the video when answering queries. However, these models face ch…
Online In-Context Distillation for Low-Resource Vision Language Models
Zhiqi Kang, Rahaf Aljundi, Vaggelis Dorovatas +1
As the field continues its push for ever more resources, this work turns the spotlight on a critical question: how can vision-language models (VLMs) be adapted to thrive in low-res…
Cross-Layer Attention Probing for Fine-Grained Hallucination Detection
Malavika Suresh, Rahaf Aljundi, Ikechukwu Nkisi-Orji +1
With the large-scale adoption of Large Language Models (LLMs) in various applications, there is a growing reliability concern due to their tendency to generate inaccurate text, i.e…
Efficient Few-Shot Continual Learning in Vision-Language Models
Aristeidis Panos, Rahaf Aljundi, Daniel Olmeda Reino +1
Vision-language models (VLMs) excel in tasks such as visual question answering and image captioning. However, VLMs are often limited by their use of pretrained image encoders, like…
Personalization Toolkit: Training Free Personalization of Large Vision Language Models
Soroush Seifi, Vaggelis Dorovatas, Matteo Cassinelli +3
Personalization of Large Vision-Language Models (LVLMs) involves customizing models to recognize specific users or object instances and to generate contextually tailored responses.…
The Phantom Menace: Unmasking Privacy Leakages in Vision-Language Models
Simone Caldarella, Massimiliano Mancini, Elisa Ricci +1
Vision-Language Models (VLMs) combine visual and textual understanding, rendering them well-suited for diverse tasks like generating image captions and answering visual questions a…
Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models
Aristeidis Panos, Rahaf Aljundi, Daniel Olmeda Reino +1
Vision language models (VLMs) demonstrate impressive capabilities in visual question answering and image captioning, acting as a crucial link between visual and language models. Ho…
Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retention
Vaibhav Singh, Rahaf Aljundi, Eugene Belilovsky
Foundational Vision-Language Models (VLMs) excel across diverse tasks, but adapting them to new domains without forgetting prior knowledge remains a critical challenge. Continual L…
Annotation Free Semantic Segmentation with Vision Foundation Models
Soroush Seifi, Daniel Olmeda Reino, Fabien Despinoy +1
Semantic Segmentation is one of the most challenging vision tasks, usually requiring large amounts of training data with expensive pixel level annotations. With the success of foun…
Incremental Object-Based Novelty Detection with Feedback Loop
Simone Caldarella, Elisa Ricci, Rahaf Aljundi
Object-based Novelty Detection (ND) aims to identify unknown objects that do not belong to classes seen during training by an object detection model. The task is particularly cruci…
OOD Aware Supervised Contrastive Learning
Soroush Seifi, Daniel Olmeda Reino, Nikolay Chumerin +1
Out-of-Distribution (OOD) detection is a crucial problem for the safe deployment of machine learning models identifying samples that fall outside of the training distribution, i.e.…