10 papers
Continual Safety Alignment via Gradient-Based Sample Selection
Thong Bach, Dung Nguyen, Thao Minh Le +1
Large language models require continuous adaptation to new tasks while preserving safety alignment. However, fine-tuning on even benign data often compromises safety behaviors, inc…
Guardrails in Logit Space: Safety Token Regularization for LLM Alignment
Thong Bach, Truyen Tran
Fine-tuning well-aligned large language models (LLMs) on new domains often degrades their safety alignment, even when using benign datasets. Existing safety alignment techniques pr…
Do Domain-specific Experts exist in MoE-based LLMs?
Giang Do, Hung Le, Truyen Tran
In the era of Large Language Models (LLMs), the Mixture of Experts (MoE) architecture has emerged as an effective approach for training extremely large models with improved computa…
Curvature-Aware Safety Restoration In LLMs Fine-Tuning
Thong Bach, Thanh Nguyen-Tang, Dung Nguyen +2
Fine-tuning Large Language Models (LLMs) for downstream tasks often compromises safety alignment, even when using parameter-efficient methods like LoRA. In this work, we uncover a…
Rethinking Deep Alignment Through The Lens Of Incomplete Learning
Thong Bach, Dung Nguyen, Thao Minh Le +1
Large language models exhibit systematic vulnerabilities to adversarial attacks despite extensive safety alignment. We provide a mechanistic analysis revealing that position-depend…
Planner-Refiner: Dynamic Space-Time Refinement for Vision-Language Alignment in Videos
Tuyen Tran, Thao Minh Le, Quang-Hung Le +1
Vision-language alignment in video must address the complexity of language, evolving interacting entities, their action chains, and semantic gaps between language and vision. This…
Towards Agentic AI for Multimodal-Guided Video Object Segmentation
Tuyen Tran, Thao Minh Le, Truyen Tran
Referring-based Video Object Segmentation is a multimodal problem that requires producing fine-grained segmentation results guided by external cues. Traditional approaches to this…
Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models
Quang-Hung Le, Long Hoang Dang, Ngan Le +2
Existing Large Vision-Language Models (LVLMs) excel at matching concepts across multi-modal inputs but struggle with compositional concepts and high-level relationships between ent…
Unified Framework with Consistency across Modalities for Human Activity Recognition
Tuyen Tran, Thao Minh Le, Hung Tran +1
Recognizing human activities in videos is challenging due to the spatio-temporal complexity and context-dependence of human interactions. Prior studies often rely on single input m…
SADL: An Effective In-Context Learning Method for Compositional Visual QA
Long Hoang Dang, Thao Minh Le, Vuong Le +2
Large vision-language models (LVLMs) offer a novel capability for performing in-context learning (ICL) in Visual QA. When prompted with a few demonstrations of image-question-answe…