9 citations · 20 across the 21 of their papers we have counts for
24 papers
Personalized Face Privacy Protection From a Single Image
Zachary Yahn, Fatih Ilhan, Tiansheng Huang +5
Photos of faces uploaded online are vulnerable to malicious actors who can scrape facial images from online sources and intrude on personal privacy via unauthorized use of facial r…
LLM-Based Data Generation and Clinical Skills Evaluation for Low-Resource French OSCEs
Tian Huang, Tom Bourgeade, Irina Illina
Objective Structured Clinical Examinations (OSCEs) are the standard method for assessing medical students' clinical and communication skills through structured patient interviews.…
Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding
Fatih Ilhan, Gaowen Liu, Ramana Rao Kompella +5
Large Vision-Language Models (VLMs) have achieved remarkable success in multi-modal reasoning, but their inference time efficiency remains a significant challenge due to the memory…
Rebellion: Noise-Robust Reasoning Training for Audio Reasoning Models
Tiansheng Huang, Virat Shejwalkar, Oscar Chang +2
Instilling reasoning capabilities in large models (LMs) using reasoning training (RT) significantly improves LMs' performances. Thus Audio Reasoning Models (ARMs), i.e., audio LMs…
FedHFT: Efficient Federated Finetuning with Heterogeneous Edge Clients
Fatih Ilhan, Selim Furkan Tekin, Tiansheng Huang +6
Fine-tuning pre-trained large language models (LLMs) has become a common practice for personalized natural language understanding (NLU) applications on downstream tasks and domain-…
Adversarial Attention Perturbations for Large Object Detection Transformers
Zachary Yahn, Selim Furkan Tekin, Fatih Ilhan +5
Adversarial perturbations are useful tools for exposing vulnerabilities in neural networks. Existing adversarial perturbation methods for object detection are either limited to att…
A Neurosymbolic Agent System for Compositional Visual Reasoning
Yichang Xu, Gaowen Liu, Ramana Rao Kompella +5
The advancement in large language models (LLMs) and large vision models has fueled the rapid progress in multi-modal vision-language reasoning capabilities. However, existing visio…
Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable
Tiansheng Huang, Sihao Hu, Fatih Ilhan +4
Safety alignment is an important procedure before the official deployment of a Large Language Model (LLM). While safety alignment has been extensively studied for LLM, there is sti…
Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agents
Selim Furkan Tekin, Fatih Ilhan, Gaowen Liu +2
The advancement of LLMs and their accessibility have triggered renewed interest in multi-agent reinforcement learning as robust and adaptive frameworks for dynamically changing env…
Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation
Tiansheng Huang, Sihao Hu, Fatih Ilhan +2
Recent research shows that Large Language Models (LLMs) are vulnerable to harmful fine-tuning attacks -- models lose their safety alignment ability after fine-tuning on a few harmf…
Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation
Yibo Wang, Tiansheng Huang, Li Shen +6
Harmful fine-tuning attack introduces significant security risks to the fine-tuning services. Main-stream defenses aim to vaccinate the model such that the later harmful fine-tunin…
H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs
Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang +4
The alignment of pre-trained LLMs continues to draw significant attention from both industry and academia, aiming to ensure responses that are helpful, harmless, and honest. Howeve…
LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity
Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang +2
Combining large language models during training or at inference time has shown substantial performance gain over component LLMs. This paper presents LLM-TOPLA, a diversity-optimize…
Data Poisoning and Leakage Analysis in Federated Learning
Wenqi Wei, Tiansheng Huang, Zachary Yahn +3
Data poisoning and leakage risks impede the massive deployment of federated learning in the real world. This chapter reveals the truths and pitfalls of understanding two dominating…
Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey
Tiansheng Huang, Sihao Hu, Fatih Ilhan +2
Recent research demonstrates that the nascent fine-tuning-as-a-service business model exposes serious safety concerns: fine-tuning with a few harmful data uploaded from the users c…
Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation
Tiansheng Huang, Sihao Hu, Fatih Ilhan +2
Harmful fine-tuning attack poses serious safety concerns for large language models' fine-tuning-as-a-service. While existing defenses have been proposed to mitigate the issue, thei…
Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning
Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi +2
Safety aligned Large Language Models (LLMs) are vulnerable to harmful fine-tuning attacks -- a few harmful data mixed in the fine-tuning dataset can break the LLMs's safety alignme…
Personalized Privacy Protection Mask Against Unauthorized Facial Recognition
Ka-Ho Chow, Sihao Hu, Tiansheng Huang +1
Face recognition (FR) can be abused for privacy intrusion. Governments, private companies, or even individual attackers can collect facial images by web scraping to build an FR sys…
Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack
Tiansheng Huang, Sihao Hu, Fatih Ilhan +2
Recent studies show that Large Language Models (LLMs) with safety alignment can be jail-broken by fine-tuning on a dataset mixed with harmful data. First time in the literature, we…
Robust Few-Shot Ensemble Learning with Focal Diversity-Based Pruning
Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang +4
This paper presents FusionShot, a focal diversity optimized few-shot ensemble learning approach for boosting the robustness and generalization performance of pre-trained few-shot m…
PokeLLMon: A Human-Parity Agent for Pokemon Battles with Large Language Models
Sihao Hu, Tiansheng Huang, Ling Liu
We introduce PokeLLMon, the first LLM-embodied agent that achieves human-parity performance in tactical battle games, as demonstrated in Pokemon battles. The design of PokeLLMon in…
A Survey on Large Language Model-Based Game Agents
Sihao Hu, Tiansheng Huang, Gaowen Liu +6
Game environments provide rich, controllable settings that stimulate many aspects of real-world complexity. As such, game agents offer a valuable testbed for exploring capabilities…
Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack
Tiansheng Huang, Sihao Hu, Ling Liu
The new paradigm of finetuning-as-a-service introduces a new attack surface for Large Language Models (LLMs): a few harmful data uploaded by users can easily trick the finetuning t…
Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives
Sihao Hu, Tiansheng Huang, Fatih İlhan +2
This paper provides a systematic analysis of the opportunities, challenges, and potential solutions of harnessing Large Language Models (LLMs) such as GPT-4 to dig out vulnerabilit…