13 papers
DistMoE: Private-data Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning
Mainak Singha, Niccolò Biondi, Elisa Ricci +1
Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centrali…
ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models
Thomas De Min, Subhankar Roy, Stéphane Lathuilière +2
Effective collaboration begins with knowing when to ask for help. For example, when trying to identify an occluded object, a human would ask someone to remove the obstruction. Can…
Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs
Michael Rottoli, Subhankar Roy, Stefano Paraboschi
Diffusion-based Large Language Models (D-LLMs) represent a promising frontier in generative AI, offering fully parallel token generation that can lead to significant throughput adv…
Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
Firas Gabetni, Giuseppe Curci, Andrea Pilzer +3
Uncertainty quantification (UQ) is essential for deploying deep neural networks in safety-critical settings. Although methods like Deep Ensembles achieve strong UQ performance, the…
Organizing Unstructured Image Collections using Natural Language
Mingxuan Liu, Zhun Zhong, Jun Li +3
In this work, we introduce and study the novel task of Open-ended Semantic Multiple Clustering (OpenSMC). Given a large, unstructured image collection, the goal is to automatically…
How (Mis)calibrated is Your Federated CLIP and What To Do About It?
Mainak Singha, Masih Aminbeidokhti, Paolo Casari +3
While vision-language models like CLIP have been extensively studied, their calibration, crucial for reliable predictions, has received limited attention. Although a few prior work…