28 citations · 44 across the 18 of their papers we have counts for
20 papers
FrameScout: Scouting Query-Relevant Frames for Long Video Understanding
Haonan Hu, Shuhao Chen, Weisen Jiang +3
Long video understanding requires VideoLLMs to answer queries about videos spanning tens of minutes to hours, but such long videos produce far more visual tokens than current Video…
SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selection
Shuhao Chen, Weisen Jiang, Yeqi Gong +5
Fine-tuning large language models often undermines their safety alignment, a problem further amplified by harmful fine-tuning attacks in which adversarial data removes safeguards a…
RxEval: A Prescription-Level Benchmark for Evaluating LLM Medication Recommendation
Shuhao Chen, Weisen Jiang, Changmiao Wang +4
Inpatient medication recommendation requires clinicians to repeatedly select specific medications, doses, and routes as a patient's condition evolves. Existing benchmarks formulate…
MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification
Weisen Jiang, Shuhao Chen, Sinno Jialin Pan
Mixture-of-Experts (MoE) models scale capacity by combining specialized experts, but most existing approaches assume centralized access to training data. In practice, data are dist…
MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generation
Weisen Jiang, Sinno Jialin Pan
This paper introduces MetaDefense, a novel framework for defending against finetuning-based jailbreak attacks in large language models (LLMs). We observe that existing defense mech…
PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
Baijiong Lin, Weisen Jiang, Yuancheng Xu +2
Multi-objective test-time alignment aims to adapt large language models (LLMs) to diverse multi-dimensional user preferences during inference while keeping LLMs frozen. Recently, G…
RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models
Shuhao Chen, Weisen Jiang, Baijiong Lin +2
Recent works show that assembling multiple off-the-shelf large language models (LLMs) can harness their complementary abilities. To achieve this, routing is a promising method, whi…
Enhancing Sharpness-Aware Minimization by Learning Perturbation Radius
Xuehao Wang, Weisen Jiang, Shuai Fu +1
Sharpness-aware minimization (SAM) is to improve model generalization by searching for flat minima in the loss landscape. The SAM update consists of one step for computing the pert…
MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoders
Baijiong Lin, Weisen Jiang, Pengguang Chen +2
Multi-task dense scene understanding, which trains a model for multiple dense prediction tasks, has a wide range of application scenarios. Capturing long-range dependency and enhan…
Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategy
Tao Li, Weisen Jiang, Fanghui Liu +2
Pre-training followed by fine-tuning is widely adopted among practitioners. The performance can be improved by "model soups"~\cite{wortsman2022model} via exploring various hyperpar…
MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoders
Baijiong Lin, Weisen Jiang, Pengguang Chen +3
Multi-task dense scene understanding, which learns a model for multiple dense prediction tasks, has a wide range of application scenarios. Modeling long-range dependency and enhanc…
GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning
Yanbin Wei, Shuai Fu, Weisen Jiang +5
Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in a textual format, they overlook the rich vi…
VLLaVO: Mitigating Visual Gap through LLMs
Shuhao Chen, Yulong Zhang, Weisen Jiang +2
Recent advances achieved by deep learning models rely on the independent and identically distributed assumption, hindering their applications in real-world scenarios with domain sh…
BYOM: Building Your Own Multi-Task Model For Free
Weisen Jiang, Baijiong Lin, Han Shi +3
Recently, various merging methods have been proposed to build a multi-task model from task-specific finetuned models without retraining. However, existing methods suffer from a lar…
Domain-Guided Conditional Diffusion Model for Unsupervised Domain Adaptation
Yulong Zhang, Shuhao Chen, Weisen Jiang +3
Limited transferability hinders the performance of deep learning models when applied to new application scenarios. Recently, Unsupervised Domain Adaptation (UDA) has achieved signi…
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
Longhui Yu, Weisen Jiang, Han Shi +7
Large language models (LLMs) have pushed the limits of natural language understanding and exhibited excellent problem-solving ability. Despite the great success, most existing open…
Dual-Balancing for Multi-Task Learning
Baijiong Lin, Weisen Jiang, Feiyang Ye +6
Multi-task learning aims to learn multiple related tasks simultaneously and has achieved great success in various fields. However, the disparity in loss and gradient scales among t…
Forward-Backward Reasoning in Large Language Models for Mathematical Verification
Weisen Jiang, Han Shi, Longhui Yu +4
Self-Consistency samples diverse reasoning chains with answers and chooses the final answer by majority voting. It is based on forward reasoning and cannot further improve performa…
Effective Structured Prompting by Meta-Learning and Representative Verbalizer
Weisen Jiang, Yu Zhang, James T. Kwok
Prompt tuning for pre-trained masked language models (MLM) has shown promising performance in natural language processing tasks with few labeled examples. It tunes a prompt for the…
An Adaptive Policy to Employ Sharpness-Aware Minimization
Weisen Jiang, Hansi Yang, Yu Zhang +1
Sharpness-aware minimization (SAM), which searches for flat minima by min-max optimization, has been shown to be useful in improving model generalization. However, since each SAM u…