70 citations · 357 across the 62 of their papers we have counts for
16 papers · 1 filter
Multimodal Commonsense Knowledge Distillation for Visual Question Answering
Shuo Yang, Siwen Luo, Soyeon Caren Han
Existing Multimodal Large Language Models (MLLMs) and Visual Language Pretrained Models (VLPMs) have shown remarkable performances in the general Visual Question Answering (VQA). H…
TriG-NER: Triplet-Grid Framework for Discontinuous Named Entity Recognition
Rina Carines Cabral, Soyeon Caren Han, Areej Alhassan +3
Discontinuous Named Entity Recognition (DNER) presents a challenging problem where entities may be scattered across multiple non-adjacent tokens, making traditional sequence labell…
'No' Matters: Out-of-Distribution Detection in Multimodality Long Dialogue
Rena Gao, Xuetong Wu, Siwen Luo +2
Out-of-distribution (OOD) detection in multimodal contexts is essential for identifying deviations in combined inputs from different modalities, particularly in applications like o…
Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
Soyeon Caren Han, Feiqi Cao, Josiah Poon +1
This tutorial explores recent advancements in multimodal pretrained and large models, capable of integrating and processing diverse data forms such as text, images, audio, and vide…
SynJAC: Synthetic-data-driven Joint-granular Adaptation and Calibration for Domain Specific Scanned Document Key Information Extraction
Yihao Ding, Soyeon Caren Han, Zechuan Li +1
Visually Rich Documents (VRDs), comprising elements such as charts, tables, and paragraphs, convey complex information across diverse domains. However, extracting key information f…
MSG-Chart: Multimodal Scene Graph for ChartQA
Yue Dai, Soyeon Caren Han, Wei Liu
Automatic Chart Question Answering (ChartQA) is challenging due to the complex distribution of chart elements with patterns of the underlying data not explicitly displayed in chart…
3M-Health: Multimodal Multi-Teacher Knowledge Distillation for Mental Health Detection
Rina Carines Cabral, Siwen Luo, Josiah Poon +1
The significance of mental health classification is paramount in contemporary society, where digital platforms serve as crucial sources for monitoring individuals' well-being. Howe…
MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU
Yan Li, So-Eon Kim, Seong-Bae Park +1
Although Large Language Models (LLMs) can generate coherent text, they often struggle to recognise user intent behind queries. In contrast, Natural Language Understanding (NLU) mod…
Deep Learning based Visually Rich Document Content Understanding: A Survey
Yihao Ding, Soyeon Caren Han, Jean Lee +1
Visually Rich Documents (VRDs) play a vital role in domains such as academia, finance, healthcare, and marketing, as they convey information through a combination of text, layout,…
3M: Multi-modal Multi-task Multi-teacher Learning for Game Event Detection
Thye Shan Ng, Feiqi Cao, Soyeon Caren Han
Esports has rapidly emerged as a global phenomenon with an ever-expanding audience via platforms, like YouTube. Due to the inherent complexity nature of the game, it is challenging…
Game-MUG: Multimodal Oriented Game Situation Understanding and Commentary Generation Dataset
Zhihao Zhang, Feiqi Cao, Yingbin Mo +3
The dynamic nature of esports makes the situation relatively complicated for average viewers. Esports broadcasting involves game expert casters, but the caster-dependent game comme…
PEACH: Pretrained-embedding Explanation Across Contextual and Hierarchical Structure
Feiqi Cao, Caren Han, Hyunsuk Chung
In this work, we propose a novel tree-based explanation technique, PEACH (Pretrained-embedding Explanation Across Contextual and Hierarchical Structure), that can explain how text-…
PDF-MVQA: A Dataset for Multimodal Information Retrieval in PDF-based Visual Question Answering
Yihao Ding, Kaixuan Ren, Jiabin Huang +2
Document Question Answering (QA) presents a challenge in understanding visually-rich documents (VRD), particularly those dominated by lengthy textual content like research journal…
A Survey of Large Language Models in Finance (FinLLMs)
Jean Lee, Nicholas Stevens, Soyeon Caren Han +1
Large Language Models (LLMs) have shown remarkable capabilities across a wide variety of Natural Language Processing (NLP) tasks and have attracted attention from multiple domains,…
3MVRD: Multimodal Multi-task Multi-teacher Visually-Rich Form Document Understanding
Yihao Ding, Lorenzo Vaiani, Caren Han +4
This paper presents a groundbreaking multimodal, multi-task, multi-teacher joint-grained knowledge distillation model for visually-rich form document understanding. The model is de…
SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling
Eileen Wang, Soyeon Caren Han, Josiah Poon
Visual storytelling aims to automatically generate a coherent story based on a given image sequence. Unlike tasks like image captioning, visual stories should contain factual descr…