5 papers
Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment
Anjila Budathoki, Manish Dhakal, Benjamin M. Ampel +1
Prior research has demonstrated that the choice of prompt template during Supervised Fine-Tuning (SFT) significantly impacts the robustness of safety alignment afterwards. However,…
DistillLens: Symmetric Knowledge Distillation Through Logit Lens
Manish Dhakal, Uthman Jinadu, Anjila Budathoki +2
Standard Knowledge Distillation (KD) compresses Large Language Models (LLMs) by optimizing final outputs, yet it typically treats the teacher's intermediate layer's thought process…
Can a Unimodal Language Agent Provide Preferences to Tune a Multimodal Vision-Language Model?
Sazia Tabasum Mim, Jack Morris, Manish Dhakal +3
To explore a more scalable path for adding multimodal capabilities to existing LLMs, this paper addresses a fundamental question: Can a unimodal LLM, relying solely on text, reason…
GFT: Graph Feature Tuning for Efficient Point Cloud Analysis
Manish Dhakal, Venkat R. Dasari, Rajshekhar Sunderraman +1
Parameter-efficient fine-tuning (PEFT) significantly reduces computational and memory costs by updating only a small subset of the model's parameters, enabling faster adaptation to…
Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation
Anjila Budathoki, Manish Dhakal
Adversarial attacks have been fairly explored for computer vision and vision-language models. However, the avenue of adversarial attack for the vision language segmentation models…