23 citations · 66 across the 16 of their papers we have counts for
16 papers · 1 filter
Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection
Jingbiao Mei, Jinghong Chen, Guangyu Yang +2
Hateful memes have become a significant concern on the Internet, necessitating robust automated detection systems. While Large Multimodal Models (LMMs) have shown promise in hatefu…
On Extending Direct Preference Optimization to Accommodate Ties
Jinghong Chen, Guangyu Yang, Weizhe Lin +2
We derive and investigate two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two wel…
Control-DAG: Constrained Decoding for Non-Autoregressive Directed Acyclic T5 using Weighted Finite State Automata
Jinghong Chen, Weizhe Lin, Jingbiao Mei +1
The Directed Acyclic Transformer is a fast non-autoregressive (NAR) model that performs well in Neural Machine Translation. Two issues prevent its application to general Natural La…
Few-Shot VQA with Frozen LLMs: A Tale of Two Approaches
Igor Sterner, Weizhe Lin, Jinghong Chen +1
Two approaches have emerged to input images into large language models (LLMs). The first is to caption images into natural language. The second is to map image feature embeddings i…
PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers
Weizhe Lin, Jingbiao Mei, Jinghong Chen +1
Large Multimodal Models (LMMs) excel in natural language and visual understanding but are challenged by exacting tasks such as Knowledge-based Visual Question Answering (KB-VQA) wh…
Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding
Guangyu Yang, Jinghong Chen, Weizhe Lin +1
Minimum Bayes Risk (MBR) decoding can significantly improve translation performance of Multilingual Large Language Models (MLLMs). However, MBR decoding is computationally expensiv…
Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning
Jingbiao Mei, Jinghong Chen, Weizhe Lin +2
Hateful memes have emerged as a significant concern on the Internet. Detecting hateful memes requires the system to jointly understand the visual and textual modalities. Our invest…
Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering
Weizhe Lin, Jinghong Chen, Jingbiao Mei +2
Knowledge-based Visual Question Answering (KB-VQA) requires VQA systems to utilize knowledge from external knowledge bases to answer visually-grounded questions. Retrieval-Augmente…
Grounding Description-Driven Dialogue State Trackers with Knowledge-Seeking Turns
Alexandru Coca, Bo-Hsiang Tseng, Jinghong Chen +4
Schema-guided dialogue state trackers can generalise to new domains without further training, yet they are sensitive to the writing style of the schemata. Augmenting the training s…
FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answering
Weizhe Lin, Zhilin Wang, Bill Byrne
The widely used Fact-based Visual Question Answering (FVQA) dataset contains visually-grounded questions that require information retrieval using common sense knowledge graphs to a…
More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase Ranking
A. Coca, B. H. Tseng, W. Lin +1
The schema-guided paradigm overcomes scalability issues inherent in building task-oriented dialogue (TOD) agents with static ontologies. Instead of operating on dialogue context al…
Schema-Guided Semantic Accuracy: Faithfulness in Task-Oriented Dialogue Response Generation
Jinghong Chen, Weizhe Lin, Bill Byrne
Ensuring that generated utterances are faithful to dialogue actions is crucial for Task-Oriented Dialogue Response Generation. Slot Error Rate (SER) only partially measures generat…
Retrieval Augmented Visual Question Answering with Outside Knowledge
Weizhe Lin, Bill Byrne
Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images. Recent OK-VQA systems…
Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking
Weizhe Lin, Bo-Hsiang Tseng, Bill Byrne
Dialogue State Tracking is central to multi-domain task-oriented dialogue systems, responsible for extracting information from user utterances. We present a novel hybrid architectu…
Learning Similarity between Movie Characters and Its Potential Implications on Understanding Human Experiences
Zhilin Wang, Weizhe Lin, Xiaodong Wu
While many different aspects of human experiences have been studied by the NLP community, none has captured its full richness. We propose a new task to capture this richness based…
Author2Vec: A Framework for Generating User Embedding
Xiaodong Wu, Weizhe Lin, Zhilin Wang +1
Online forums and social media platforms provide noisy but valuable data every day. In this paper, we propose a novel end-to-end neural network-based user embedding system, Author2…