6 papers
Do Language Models Track Entities Across State Changes?
Zilu Tang, Qiao Zhao, Gabriel Franco +4
Entity tracking (ET), the ability to keep track of states, is a fundamental skill that underlies complex reasoning. An increasing amount of work investigates how transformer langua…
Beyond Transfer Accuracy: Faithful Circuits for Controlled Low-Resource Adaptation
Khumaisa Nur'aini, Ayu Purwarianti, Alham Fikri Aji +1
Existing circuit discovery methods rely on templated tasks with clean counterfactuals, limiting their use on diverse natural text. We adapt Contextual Decomposition for Transformer…
mR3: Multilingual Rubric-Agnostic Reward Reasoning Models
David Anugraha, Shou-Yi Hung, Zilu Tang +3
Evaluation using Large Language Model (LLM) judges has been widely adopted in English and shown to be effective for automatic evaluation. However, their performance does not genera…
R3: Robust Rubric-Agnostic Reward Models
David Anugraha, Zilu Tang, Lester James V. Miranda +5
Reward models are essential for aligning language model outputs with human preferences, yet existing approaches often lack both controllability and interpretability. These models a…
Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability
Genta Indra Winata, David Anugraha, Emmy Liu +17
High-quality datasets are fundamental to training and evaluating machine learning models, yet their creation-especially with accurate human annotations-remains a significant challe…
Do Language Models Understand Honorific Systems in Javanese?
Mohammad Rifqi Farhansyah, Iwan Darmawan, Adryan Kusumawardhana +3
The Javanese language features a complex system of honorifics that vary according to the social status of the speaker, listener, and referent. Despite its cultural and linguistic s…