9 citations · 19 across the 22 of their papers we have counts for
25 papers
Alliance of Stochastic Routing and Reinforcement Fusion for Multi-Model Reasoning
Selim Furkan Tekin, Ling Liu
This paper studies dynamic model routing under uncertainty in multimodal settings, where a system must select and potentially compose multiple models to answer a query. Standard re…
Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding
Fatih Ilhan, Gaowen Liu, Ramana Rao Kompella +5
Large Vision-Language Models (VLMs) have achieved remarkable success in multi-modal reasoning, but their inference time efficiency remains a significant challenge due to the memory…
A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoning
Yichang Xu, Gaowen Liu, Ramana Rao Kompella +6
This paper presents a multi-agent perception-action exploration alliance, dubbed A4VL, for efficient long-video reasoning. A4VL operates in a multi-round perception-action explorat…
Vision Verification Enhanced Fusion of VLMs for Efficient Visual Reasoning
Selim Furkan Tekin, Yichang Xu, Gaowen Liu +3
With the growing number and diversity of Vision-Language Models (VLMs), many works explore language-based ensemble, collaboration, and routing techniques across multiple VLMs to im…
MELT: A Behavioral Trace Dataset for High-Risk Memecoin Launch Detection
Sihao Hu, Selim Furkan Tekin, Yichang Xu +1
Launchpads have become the dominant mechanism for issuing memecoins, exposing investors to a new class of high-risk launches that existing rug-pull detection methods cannot capture…
Large Language Model based Smart Contract Auditing with LLMBugScanner
Yining Yuan, Yifei Wang, Yichang Xu +3
This paper presents LLMBugScanner, a large language model (LLM) based framework for smart contract vulnerability detection using fine-tuning and ensemble learning. Smart contract a…
Rebellion: Noise-Robust Reasoning Training for Audio Reasoning Models
Tiansheng Huang, Virat Shejwalkar, Oscar Chang +2
Instilling reasoning capabilities in large models (LMs) using reasoning training (RT) significantly improves LMs' performances. Thus Audio Reasoning Models (ARMs), i.e., audio LMs…
FedHFT: Efficient Federated Finetuning with Heterogeneous Edge Clients
Fatih Ilhan, Selim Furkan Tekin, Tiansheng Huang +6
Fine-tuning pre-trained large language models (LLMs) has become a common practice for personalized natural language understanding (NLU) applications on downstream tasks and domain-…
Adversarial Attention Perturbations for Large Object Detection Transformers
Zachary Yahn, Selim Furkan Tekin, Fatih Ilhan +5
Adversarial perturbations are useful tools for exposing vulnerabilities in neural networks. Existing adversarial perturbation methods for object detection are either limited to att…
A Neurosymbolic Agent System for Compositional Visual Reasoning
Yichang Xu, Gaowen Liu, Ramana Rao Kompella +5
The advancement in large language models (LLMs) and large vision models has fueled the rapid progress in multi-modal vision-language reasoning capabilities. However, existing visio…
Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable
Tiansheng Huang, Sihao Hu, Fatih Ilhan +4
Safety alignment is an important procedure before the official deployment of a Large Language Model (LLM). While safety alignment has been extensively studied for LLM, there is sti…
Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agents
Selim Furkan Tekin, Fatih Ilhan, Gaowen Liu +2
The advancement of LLMs and their accessibility have triggered renewed interest in multi-agent reinforcement learning as robust and adaptive frameworks for dynamically changing env…
Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation
Tiansheng Huang, Sihao Hu, Fatih Ilhan +2
Recent research shows that Large Language Models (LLMs) are vulnerable to harmful fine-tuning attacks -- models lose their safety alignment ability after fine-tuning on a few harmf…
H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs
Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang +4
The alignment of pre-trained LLMs continues to draw significant attention from both industry and academia, aiming to ensure responses that are helpful, harmless, and honest. Howeve…
LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity
Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang +2
Combining large language models during training or at inference time has shown substantial performance gain over component LLMs. This paper presents LLM-TOPLA, a diversity-optimize…
Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey
Tiansheng Huang, Sihao Hu, Fatih Ilhan +2
Recent research demonstrates that the nascent fine-tuning-as-a-service business model exposes serious safety concerns: fine-tuning with a few harmful data uploaded from the users c…
Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation
Tiansheng Huang, Sihao Hu, Fatih Ilhan +2
Harmful fine-tuning attack poses serious safety concerns for large language models' fine-tuning-as-a-service. While existing defenses have been proposed to mitigate the issue, thei…
Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning
Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi +2
Safety aligned Large Language Models (LLMs) are vulnerable to harmful fine-tuning attacks -- a few harmful data mixed in the fine-tuning dataset can break the LLMs's safety alignme…
Personalized Privacy Protection Mask Against Unauthorized Facial Recognition
Ka-Ho Chow, Sihao Hu, Tiansheng Huang +1
Face recognition (FR) can be abused for privacy intrusion. Governments, private companies, or even individual attackers can collect facial images by web scraping to build an FR sys…
Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack
Tiansheng Huang, Sihao Hu, Fatih Ilhan +2
Recent studies show that Large Language Models (LLMs) with safety alignment can be jail-broken by fine-tuning on a dataset mixed with harmful data. First time in the literature, we…
Robust Few-Shot Ensemble Learning with Focal Diversity-Based Pruning
Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang +4
This paper presents FusionShot, a focal diversity optimized few-shot ensemble learning approach for boosting the robustness and generalization performance of pre-trained few-shot m…
PokeLLMon: A Human-Parity Agent for Pokemon Battles with Large Language Models
Sihao Hu, Tiansheng Huang, Ling Liu
We introduce PokeLLMon, the first LLM-embodied agent that achieves human-parity performance in tactical battle games, as demonstrated in Pokemon battles. The design of PokeLLMon in…
A Survey on Large Language Model-Based Game Agents
Sihao Hu, Tiansheng Huang, Gaowen Liu +6
Game environments provide rich, controllable settings that stimulate many aspects of real-world complexity. As such, game agents offer a valuable testbed for exploring capabilities…
Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack
Tiansheng Huang, Sihao Hu, Ling Liu
The new paradigm of finetuning-as-a-service introduces a new attack surface for Large Language Models (LLMs): a few harmful data uploaded by users can easily trick the finetuning t…