14 papers
One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation
Hyunjong Ok, Minjae Park, Jaeho Lee
Reordering multiple-choice options can change a model's answer, an effect studied as selection bias (a preference for certain option letters) and as order sensitivity (changes in a…
Hierarchical Compression of Vision-Language Model Benchmarks
Hyunjong Ok, Seunggu Kang, Jaeho Lee
Thorough evaluation of vision-language models (VLMs) has become prohibitively expensive, as benchmarks span an ever-broader spectrum of capabilities and new models arrive at a rele…
OmniKVQuant: KV Cache Quantization for Omni-LLMs
Suho Yoo, Hyunjong Ok, Jongmin Choi +2
As Omni-modal large language models (Omni-LLMs) take in audio, video and text together, their KV cache memory cost grows. KV cache quantization is the de facto approach in text-onl…
Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models
Hyunjong Ok, Jaeho Lee
Large language models exhibit surprising sensitivity to the structure of the prompt, but the mechanisms underlying this sensitivity remain poorly understood. In this work, we condu…
Understanding the Effects of Distractors on Reasoning Vision-Language Models
Jiyun Bae, Hyunjong Ok, Sangwoo Mo +1
How does irrelevant information (i.e., distractors) affect test-time scaling in vision-language models (VLMs)? Prior work on text-only language models has shown that textual distra…
AuditoryBench++: Can Language Models Understand Auditory Knowledge without Hearing?
Hyunjong Ok, Suho Yoo, Hyeonjun Kim +1
Even without directly hearing sounds, humans can effortlessly reason about auditory properties, such as pitch, loudness, or sound-source associations, drawing on auditory commonsen…
TempCore: Are Video QA Benchmarks Temporally Grounded?
Hyunjong Ok, Jaeho Lee
Vision-language models (VLMs) can ingest only a limited number of video frames, making frame selection a practical necessity. But do current Video QA benchmarks genuinely require t…
Speculative End-Turn Detector for Efficient Speech Chatbot Assistant
Hyunjong Ok, Suho Yoo, Jaeho Lee
Spoken dialogue systems powered by large language models have demonstrated remarkable abilities in understanding human speech and generating appropriate spoken responses. However,…
Imagine to Hear: Auditory Knowledge Generation can be an Effective Assistant for Language Models
Suho Yoo, Hyunjong Ok, Jaeho Lee
Language models pretrained on text-only corpora often struggle with tasks that require auditory commonsense knowledge. Previous work addresses this problem by augmenting the langua…
S2Cap: A Benchmark and a Baseline for Singing Style Captioning
Hyunjong Ok, Jaeho Lee
Singing voices contain much richer information than common voices, including varied vocal and acoustic properties. However, current open-source audio-text datasets for singing voic…
AudioBERT: Audio Knowledge Augmented Language Model
Hyunjong Ok, Suho Yoo, Jaeho Lee
Recent studies have identified that language models, pretrained on text-only datasets, often lack elementary visual knowledge, \textit{e.g.,} colors of everyday objects. Motivated…
Decoding with Limited Teacher Supervision Requires Understanding When to Trust the Teacher
Hyunjong Ok, Jegwang Ryu, Jaeho Lee
How can small-scale large language models (LLMs) efficiently utilize the supervision of LLMs to improve their generative quality? This question has been well studied in scenarios w…
SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entities
Hyunjong Ok, Taeho Kil, Sukmin Seo +1
Recent advances in named entity recognition (NER) have pushed the boundary of the task to incorporate visual signals, leading to many variants, including multi-modal NER (MNER) or…
FinTree: Financial Dataset Pretrain Transformer Encoder for Relation Extraction
Hyunjong Ok
We present FinTree, Financial Dataset Pretrain Transformer Encoder for Relation Extraction. Utilizing an encoder language model, we further pretrain FinTree on the financial datase…