5 papers · 1 filter
ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models
Duy M. H. Nguyen, Nghiem T. Diep, Trung Q. Nguyen +10
State-of-the-art medical multi-modal LLMs (med-MLLMs), such as LLaVA-Med and BioMedGPT, primarily depend on scaling model size and data volume, with training driven largely by auto…
S-Chain: Structured Visual Chain-of-Thought For Medicine
Khai Le-Duc, Duy M. H. Nguyen, Phuong T. H. Trinh +21
Faithful reasoning in medical vision-language models (VLMs) requires not only accurate predictions but also transparent alignment between textual rationales and visual evidence. Wh…
On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Nghiem T. Diep, Huy Nguyen, Chau Nguyen +5
The LLaMA-Adapter has recently emerged as an efficient fine-tuning technique for LLaMA models, leveraging zero-initialized attention to stabilize training and enhance performance.…
Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling
Phuc Minh Nguyen, Ngoc-Hieu Nguyen, Duy H. M. Nguyen +5
Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO) have emerged as alternatives to the standard Reinforcement Learning from Human Feedback (RLHF) for a…
Accelerating Transformers with Spectrum-Preserving Token Merging
Hoai-Chau Tran, Duy M. H. Nguyen, Duy M. Nguyen +7
Increasing the throughput of the Transformer architecture, a foundational component used in numerous state-of-the-art models for vision and language tasks (e.g., GPT, LLaVa), is an…