7 papers · 1 filter
CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia
Bryan Chen Zhengyu Tan, Weihua Zheng, Thong T. Doan +30
Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common use case: users seeking pract…
PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino
Jann Railey Montalan, David Demitri Africa, Jimson Paulo Layacan +3
Large language models (LLMs) process text as sequences of subword tokens, which can obscure the character-level and morphological structure that underlies word formation. This limi…
BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models
Thura Aung, Jann Railey Montalan, Jian Gang Ngui +1
We introduce BURMESE-SAN, the first holistic benchmark that systematically evaluates large language models (LLMs) for Burmese across three core NLP competencies: understanding (NLU…
SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?
Wuttikorn Ponwitayarat, Peerat Limkonchotiwat, Raymond Ng +9
Multilingual text embeddings are often assumed to encode meaning in a perspective-independent semantic space, yielding stable similarity judgments across tasks and languages. Our r…
Batayan: A Filipino NLP benchmark for evaluating Large Language Models
Jann Railey Montalan, Jimson Paulo Layacan, David Demitri Africa +5
Recent advances in large language models (LLMs) have demonstrated remarkable capabilities on widely benchmarked high-resource languages. However, linguistic nuances of under-resour…
SEA-HELM: Southeast Asian Holistic Evaluation of Language Models
Yosephine Susanto, Adithya Venkatadri Hulagadri, Jann Railey Montalan +7
With the rapid emergence of novel capabilities in Large Language Models (LLMs), the need for rigorous multilingual and multicultural benchmarks that are integrated has become more…