23 citations · 66 across the 16 of their papers we have counts for
20 papers
Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language Models
Guangyu Yang, Jinghong Chen, Jingbiao Mei +2
Large Language Models (LLMs) remain vulnerable to jailbreak attacks, which attempt to elicit harmful responses from LLMs. The evolving nature and diversity of these attacks pose ma…
Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection
Jingbiao Mei, Jinghong Chen, Guangyu Yang +2
Hateful memes have become a significant concern on the Internet, necessitating robust automated detection systems. While Large Multimodal Models (LMMs) have shown promise in hatefu…
On Extending Direct Preference Optimization to Accommodate Ties
Jinghong Chen, Guangyu Yang, Weizhe Lin +2
We derive and investigate two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two wel…
Control-DAG: Constrained Decoding for Non-Autoregressive Directed Acyclic T5 using Weighted Finite State Automata
Jinghong Chen, Weizhe Lin, Jingbiao Mei +1
The Directed Acyclic Transformer is a fast non-autoregressive (NAR) model that performs well in Neural Machine Translation. Two issues prevent its application to general Natural La…
Few-Shot VQA with Frozen LLMs: A Tale of Two Approaches
Igor Sterner, Weizhe Lin, Jinghong Chen +1
Two approaches have emerged to input images into large language models (LLMs). The first is to caption images into natural language. The second is to map image feature embeddings i…
PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers
Weizhe Lin, Jingbiao Mei, Jinghong Chen +1
Large Multimodal Models (LMMs) excel in natural language and visual understanding but are challenged by exacting tasks such as Knowledge-based Visual Question Answering (KB-VQA) wh…
Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding
Guangyu Yang, Jinghong Chen, Weizhe Lin +1
Minimum Bayes Risk (MBR) decoding can significantly improve translation performance of Multilingual Large Language Models (MLLMs). However, MBR decoding is computationally expensiv…
Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning
Jingbiao Mei, Jinghong Chen, Weizhe Lin +2
Hateful memes have emerged as a significant concern on the Internet. Detecting hateful memes requires the system to jointly understand the visual and textual modalities. Our invest…
Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering
Weizhe Lin, Jinghong Chen, Jingbiao Mei +2
Knowledge-based Visual Question Answering (KB-VQA) requires VQA systems to utilize knowledge from external knowledge bases to answer visually-grounded questions. Retrieval-Augmente…
Grounding Description-Driven Dialogue State Trackers with Knowledge-Seeking Turns
Alexandru Coca, Bo-Hsiang Tseng, Jinghong Chen +4
Schema-guided dialogue state trackers can generalise to new domains without further training, yet they are sensitive to the writing style of the schemata. Augmenting the training s…
FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answering
Weizhe Lin, Zhilin Wang, Bill Byrne
The widely used Fact-based Visual Question Answering (FVQA) dataset contains visually-grounded questions that require information retrieval using common sense knowledge graphs to a…
More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase Ranking
A. Coca, B. H. Tseng, W. Lin +1
The schema-guided paradigm overcomes scalability issues inherent in building task-oriented dialogue (TOD) agents with static ontologies. Instead of operating on dialogue context al…
Schema-Guided Semantic Accuracy: Faithfulness in Task-Oriented Dialogue Response Generation
Jinghong Chen, Weizhe Lin, Bill Byrne
Ensuring that generated utterances are faithful to dialogue actions is crucial for Task-Oriented Dialogue Response Generation. Slot Error Rate (SER) only partially measures generat…
Retrieval Augmented Visual Question Answering with Outside Knowledge
Weizhe Lin, Bill Byrne
Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images. Recent OK-VQA systems…
Transformer-Empowered Content-Aware Collaborative Filtering
Weizhe Lin, Linjun Shou, Ming Gong +4
Knowledge graph (KG) based Collaborative Filtering is an effective approach to personalizing recommendation systems for relatively static domains such as movies and books, by lever…
Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking
Weizhe Lin, Bo-Hsiang Tseng, Bill Byrne
Dialogue State Tracking is central to multi-domain task-oriented dialogue systems, responsible for extracting information from user utterances. We present a novel hybrid architectu…
Message-Aware Graph Attention Networks for Large-Scale Multi-Robot Path Planning
Qingbiao Li, Weizhe Lin, Zhe Liu +1
The domains of transport and logistics are increasingly relying on autonomous mobile robots for the handling and distribution of passengers or resources. At large system scales, fi…
Learning Similarity between Movie Characters and Its Potential Implications on Understanding Human Experiences
Zhilin Wang, Weizhe Lin, Xiaodong Wu
While many different aspects of human experiences have been studied by the NLP community, none has captured its full richness. We propose a new task to capture this richness based…
Looking At The Body: Automatic Analysis of Body Gestures and Self-Adaptors in Psychological Distress
Weizhe Lin, Indigo Orton, Qingbiao Li +2
Psychological distress is a significant and growing issue in society. Automatic detection, assessment, and analysis of such distress is an active area of research. Compared to moda…
Author2Vec: A Framework for Generating User Embedding
Xiaodong Wu, Weizhe Lin, Zhilin Wang +1
Online forums and social media platforms provide noisy but valuable data every day. In this paper, we propose a novel end-to-end neural network-based user embedding system, Author2…