4 papers
KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context
Nahyun Lee, Guijin Son, Hyunwoo Ko +4
We introduce KMMMU, a native Korean benchmark for evaluating multimodal understanding in Korean cultural and institutional settings. KMMMU contains 3,466 questions from exams nativ…
Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options
Nahyun Lee, Guijin Son
Multiple choice evaluation is widely used for benchmarking large language models, yet near ceiling accuracy in low option settings can be sustained by shortcut strategies that obsc…
KAIO: A Collection of More Challenging Korean Questions
Nahyun Lee, Guijin Son, Hyunwoo Ko +1
With the advancement of mid/post-training techniques, LLMs are pushing their boundaries at an accelerated pace. Legacy benchmarks saturate quickly (e.g., broad suites like MMLU ove…
Controlling Language Confusion in Multilingual LLMs
Nahyun Lee, Yeongseo Woo, Hyunwoo Ko +1
Large language models often suffer from language confusion, a phenomenon in which responses are partially or entirely generated in unintended languages. This critically degrades th…