147 citations · 238 across the 33 of their papers we have counts for
33 papers
Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models
Tinghui Zhu, Qin Liu, Fei Wang +2
Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities for capturing and reasoning over multimodal inputs. However, these models are prone to parametric kno…
Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models
Fei Wang, Ninareh Mehrabi, Palash Goyal +3
Data is a crucial element in large language model (LLM) alignment. Recent studies have explored using LLMs for efficient data collection. However, LLM-generated data often suffers…
Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models
Fei Wang, Xingchen Wan, Ruoxi Sun +2
Retrieval augmented generation (RAG), while effectively integrating external knowledge to address the inherent limitations of large language models (LLMs), can be hindered by imper…
Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
Qin Liu, Wenjie Mo, Terry Tong +4
The advancement of Large Language Models (LLMs) has significantly impacted various domains, including Web search, healthcare, and software development. However, as these models sca…
MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding
Fei Wang, Xingyu Fu, James Y. Huang +18
We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tas…
From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
Nan Xu, Fei Wang, Sheng Zhang +2
Motivated by in-context learning (ICL) capabilities of Large Language Models (LLMs), multimodal LLMs with additional visual modality are also exhibited with similar ICL abilities w…
FamiCom: Further Demystifying Prompts for Language Models with Task-Agnostic Performance Estimation
Bangzheng Li, Ben Zhou, Xingyu Fu +3
Language models have shown impressive in-context-learning capabilities, which allow them to benefit from input prompts and perform better on downstream end tasks. Existing works in…
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Jiangshu Du, Yibo Wang, Wenting Zhao +37
This work is motivated by two key trends. On one hand, large language models (LLMs) have shown remarkable versatility in various generative tasks such as writing, drawing, and ques…
mDPO: Conditional Preference Optimization for Multimodal Large Language Models
Fei Wang, Wenxuan Zhou, James Y. Huang +4
Direct preference optimization (DPO) has shown to be an effective method for large language model (LLM) alignment. Recent works have attempted to apply DPO to multimodal scenarios…
Offset Unlearning for Large Language Models
James Y. Huang, Wenxuan Zhou, Fei Wang +4
Despite the strong capabilities of Large Language Models (LLMs) to acquire knowledge from their training corpora, the memorization of sensitive information in the corpora such as c…
From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification
Fei Wang, Chao Shang, Sarthak Jain +6
User alignment is crucial for adapting general-purpose language models (LMs) to downstream tasks, but human annotations are often not available for all types of instructions, espec…
Monotonic Paraphrasing Improves Generalization of Language Model Prompting
Qin Liu, Fei Wang, Nan Xu +3
Performance of large language models (LLMs) may vary with different prompts or instructions of even the same task. One commonly recognized factor for this phenomenon is the model's…
Privacy-Preserving Language Model Inference with Instance Obfuscation
Yixiang Yao, Fei Wang, Srivatsan Ravi +1
Language Models as a Service (LMaaS) offers convenient access for developers and researchers to perform inference using pre-trained language models. Nonetheless, the input data and…
Contrastive Instruction Tuning
Tianyi Lorena Yan, Fei Wang, James Y. Huang +5
Instruction tuning has been used as a promising approach to improve the performance of large language models (LLMs) on unseen tasks. However, current LLMs exhibit limited robustnes…
Instructional Fingerprinting of Large Language Models
Jiashu Xu, Fei Wang, Mingyu Derek Ma +3
The exorbitant cost of training Large language models (LLMs) from scratch makes it essential to fingerprint the models to protect intellectual property via ownership authentication…
Rethinking Tabular Data Understanding with Large Language Models
Tianyang Liu, Fei Wang, Muhao Chen
Large Language Models (LLMs) have shown to be capable of various tasks, yet their capability in interpreting and reasoning over tabular data remains an underexplored area. In this…
Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking
Nan Xu, Fei Wang, Ben Zhou +3
While large language models (LLMs) have demonstrated increasing power, they have also given rise to a wide range of harmful behaviors. As representatives, jailbreak attacks can pro…
Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?
Bangzheng Li, Ben Zhou, Fei Wang +3
Despite the recent advancement in large language models (LLMs) and their high performances across numerous benchmarks, recent research has unveiled that LLMs suffer from hallucinat…
Dense Retrieval as Indirect Supervision for Large-space Decision Making
Nan Xu, Fei Wang, Mingtao Dong +1
Many discriminative natural language understanding (NLU) tasks have large label spaces. Learning such a process of large-space decision making is particularly challenging due to th…
Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer
Fei Wang, Kuan-Hao Huang, Kai-Wei Chang +1
Zero-shot cross-lingual transfer is a central task in multilingual NLP, allowing models trained in languages with more sufficient training resources to generalize to other low-reso…
Robust Natural Language Understanding with Residual Attention Debiasing
Fei Wang, James Y. Huang, Tianyi Yan +2
Natural language understanding (NLU) models often suffer from unintended dataset biases. Among bias mitigation methods, ensemble-based debiasing methods, especially product-of-expe…
Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality
Tanay Dixit, Fei Wang, Muhao Chen
Improving factual consistency of abstractive summarization has been a widely studied topic. However, most of the prior works on training factuality-aware models have ignored the ne…
From Shortcuts to Triggers: Backdoor Defense with Denoised PoE
Qin Liu, Fei Wang, Chaowei Xiao +1
Language models are often at risk of diverse backdoor attacks, especially data poisoning. Thus, it is important to investigate defense solutions for addressing them. Existing backd…
Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models
Jiashu Xu, Mingyu Derek Ma, Fei Wang +2
We investigate security concerns of the emergent instruction tuning paradigm, that models are trained on crowdsourced datasets with task instructions to achieve superior performanc…