9 papers
CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia
Bryan Chen Zhengyu Tan, Weihua Zheng, Thong T. Doan +30
Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common use case: users seeking pract…
Solar Open 2 Technical Report
Sungrae Park, Sanghoon Kim, Gyoungjin Gim +50
We present Solar Open 2, a 250B-A15B Mixture-of-Experts language model built for long-horizon agentic tasks, scaled up from Solar Open 1 (Solar Open 100B). To hold entire agent tra…
SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?
Hwiwon Lee, Jiawei Liu, Dongjun Kim +4
Finding a real vulnerability in complicated systems is a challenging, long-horizon task that demands reasoning across an entire codebase to produce a working proof-of-concept (PoC)…
LANGSAE EDITING: Improving Multilingual Information Retrieval via Post-hoc Language Identity Removal
Dongjun Kim, Jeongho Yoon, Chanjun Park +1
Dense retrieval in multilingual settings often searches over mixed-language collections, yet multilingual embeddings encode language identity alongside semantics. This language sig…
KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models
Dongjun Kim, Chanhee Park, Chanjun Park +1
The instruction-following capabilities of large language models (LLMs) are pivotal for numerous applications, from conversational agents to complex reasoning systems. However, curr…
MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation
Weihua Zheng, Zhengyuan Liu, Tanmoy Chakraborty +32
Large language models (LLMs) are now used worldwide, yet their multimodal understanding and reasoning often degrade outside Western, high-resource settings. We propose MMA-ASIA, a…
Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks
Dongjun Kim, Gyuho Shim, Yongchan Chun +3
Large Language Models are commonly judged by their scores on standard benchmarks, yet such scores often overstate real capability since they mask the mix of skills a task actually…
Enhancing Automatic Term Extraction with Large Language Models via Syntactic Retrieval
Yongchan Chun, Minhyuk Kim, Dongjun Kim +2
Automatic Term Extraction (ATE) identifies domain-specific expressions that are crucial for downstream tasks such as machine translation and information retrieval. Although large l…
Exploring Coding Spot: Understanding Parametric Contributions to LLM Coding Performance
Dongjun Kim, Minhyuk Kim, YongChan Chun +2
Large Language Models (LLMs) have demonstrated notable proficiency in both code generation and comprehension across multiple programming languages. However, the mechanisms underlyi…