16 papers
Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space
Long Minh Bui, Tuan Anh Le Van, Tung Phi Duc +3
Model merging aims to combine existing single-task solutions into a multi-task solution without additional data-driven fine-tuning.~Most existing approaches achieve this using geom…
Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm
Trong Khiem Tran, Anh Duc Chu, Quang Hung Pham +2
Cross-modal knowledge distillation (CMKD) studies how a (large) teacher model trained on one type of data (e.g., images) can guide a (smaller) student model building on another typ…
Black-Box Optimization From Small Offline Datasets via Meta Learning with Synthetic Tasks
Azza Fadhel, The Hung Tran, Trong Nghia Hoang +1
We consider the problem of offline black-box optimization, where the goal is to discover optimal designs (e.g., molecules or materials) from past experimental data. A key challenge…
Diffusion-Inspired Reconfiguration of Transformers for Uncertainty Calibration
Manh Cuong Dao, Quang Hung Pham, Phi Le Nguyen +3
Uncertainty calibration in pre-trained transformers is critical for their reliable deployment in risk-sensitive applications. Yet, most existing pre-trained transformers do not hav…
Rethinking Cross-Modal Fine-Tuning: Optimizing the Interaction Between Feature Alignment and Target Fitting
Trong Khiem Tran, Manh Cuong Dao, Phi Le Nguyen +2
Adapting pre-trained models to unseen feature modalities has become increasingly important due to the growing need for cross-disciplinary knowledge integration. A key challenge her…
Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data
Thu Hang Phung, Duong M. Nguyen, Thanh Trung Huynh +3
This paper introduces a generalized federated prompt-tuning framework for practical scenarios where local datasets are multi-modal and exhibit different distributional patterns of…