33 citations · 139 across the 28 of their papers we have counts for
33 papers
Multimodal-SEA: Benchmarking AI Agent Security under Multimodal Social Engineering Attacks
Zhitao He, Chi Him Zhang, Chengru Han +1
Personal AI agents face social engineering attacks supported by forged multimodal content, but existing evaluations mainly focus on text-based dialogue, leaving their intrinsic sec…
Before Agent Tells The Lie: Has Deception Already Been Represented?
Xinling Li, Dadi Guo, Qingyu Liu +5
Large language model (LLM)-based agents can exhibit deceptive behavior during task execution, including hiding failures, fabricating results, or falsely signaling task completion.…
ATTRICITE: Training an Open 4B Model for Citation Recovery toward Faithful Attribution
Yee Man Choi, Xuehang Guo, Songcheng Cai +3
Faithful citation attribution begins with identifying the intended source for a scientific claim. We study this source-identification capability through citation recovery: recoveri…
AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints
Jiayu Liu, Cheng Qian, Zhenhailong Wang +10
Planning for real-world problems by language models often involves both world and user constraints, which may not be fully specified upfront and are progressively disclosed through…
Veri-R1: Toward Precise and Faithful Claim Verification via Online Reinforcement Learning
Qi He, Cheng Qian, Xiusi Chen +3
Claim verification with large language models (LLMs) has recently attracted growing attention, due to their strong reasoning capabilities and transparent verification processes com…
EMCompress: Video-LLMs with Endomorphic Multimodal Compression
Zheyu Fan, Jiateng Liu, Yuji Zhang +4
Video-LLMs face a fundamental tension in long-video reasoning: static, sparse frame sampling either dilutes evidence across task-irrelevant segments at significant cost or misses f…
The Law of Knowledge Overshadowing: Towards Understanding, Predicting, and Preventing LLM Hallucination
Yuji Zhang, Sha Li, Cheng Qian +8
Hallucination is a persistent challenge in large language models (LLMs), where even with rigorous quality control, models often generate distorted facts. This paradox, in which err…
Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks
Jiayi He, Hehai Lin, Qingyun Wang +2
While Vision-Language Models (VLMs) have shown remarkable abilities in visual and language reasoning tasks, they invariably generate flawed responses. Self-correction that instruct…
Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models
Yuji Zhang, Sha Li, Jiateng Liu +5
Hallucination is often regarded as a major impediment for using large language models (LLMs), especially for knowledge-intensive tasks. Even when the training corpus consists solel…
MACAROON: Training Vision-Language Models To Be Your Engaged Partners
Shujin Wu, Yi R. Fung, Sha Li +3
Large vision-language models (LVLMs), while proficient in following instructions and responding to diverse questions, invariably generate detailed responses even when questions are…
From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models
Kung-Hsiang Huang, Hou Pong Chan, Yi R. Fung +5
Data visualization in the form of charts plays a pivotal role in data analysis, offering critical insights and aiding in informed decision-making. Automatic chart understanding has…
Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking
Yi Fung, Ruining Zhao, Jae Doo +2
Pretrained large language models have revolutionized many applications but still face challenges related to cultural bias and a lack of cultural commonsense knowledge crucial for g…
LEMMA: Towards LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation
Keyang Xuan, Li Yi, Fan Yang +3
The rise of multimodal misinformation on social platforms poses significant challenges for individuals and societies. Its increased credibility and broader impact compared to textu…
Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinement
Chenkai Sun, Ke Yang, Revanth Gangi Reddy +5
The increasing demand for personalized interactions with large language models (LLMs) calls for methodologies capable of accurately and efficiently identifying user opinions and pr…
If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents
Ke Yang, Jiateng Liu, John Wu +9
The prominent large language models (LLMs) of today differ from past language models not only in size, but also in the fact that they are trained on a combination of natural langua…
Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning
Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan +5
Recent advancements in large vision-language models (LVLMs) have led to significant progress in generating natural language descriptions for visual content and thus enhancing vario…
R-Tuning: Instructing Large Language Models to Say `I Don't Know'
Hanning Zhang, Shizhe Diao, Yong Lin +6
Large language models (LLMs) have revolutionized numerous domains with their impressive performance but still face their challenges. A predominant issue is the propensity for these…
Defining a New NLP Playground
Sha Li, Chi Han, Pengfei Yu +8
The recent explosion of performance of large language models (LLMs) has changed the field of Natural Language Processing (NLP) more abruptly and seismically than any other shift in…
Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting
Chenkai Sun, Jinning Li, Yi R. Fung +4
Automatic response forecasting for news media plays a crucial role in enabling content producers to efficiently predict the impact of news releases and prevent unexpected negative…
CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets
Lifan Yuan, Yangyi Chen, Xingyao Wang +3
Large language models (LLMs) are often augmented with tools to solve complex tasks. By generating code snippets and executing them through task-specific Application Programming Int…
Enhanced Chart Understanding in Vision and Language Task via Cross-modal Pre-training on Plot Table Pairs
Mingyang Zhou, Yi R. Fung, Long Chen +3
Building cross-model intelligence that can understand charts and communicate the salient information hidden behind them is an appealing challenge in the vision and language(V+L) co…
CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Cheng Qian, Chi Han, Yi R. Fung +3
Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning, particul…
Word Embeddings Are Steers for Language Models
Chi Han, Jialiang Xu, Manling Li +5
Language models (LMs) automatically learn word embeddings during pre-training on language corpora. Although word embeddings are usually interpreted as feature vectors for individua…
Tool Learning with Foundation Models
Yujia Qin, Shengding Hu, Yankai Lin +38
Humans possess an extraordinary ability to create and utilize tools, allowing them to overcome physical limitations and explore new frontiers. With the advent of foundation models,…