7 papers
From Token Probabilities to Semantic Constraints: Towards Declarative Probabilistic Evaluation of Language Models
Kyle Richardson, Cullen Anderson, Pranav Balakrishnan +4
While Large Language Models have improved rapidly, many fundamental questions remain about how to evaluate the knowledge and reasoning abilities they acquire, and how such evaluati…
RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering
Marisa Hudspeth, Patrick J. Burns, Brendan O'Connor
We introduce a benchmark dataset for question answering and translation in bilingual Latin and English settings, containing about 7,800 question-answer pairs. The questions are dra…
Contextual morphologically-guided tokenization for Latin encoder models
Marisa Hudspeth, Patrick J. Burns, Brendan O'Connor
Tokenization is a critical component of language model pretraining, yet standard tokenization methods often prioritize information-theoretical goals like high compression and low f…
Evaluating Morphological Alignment of Tokenizers in 70 Languages
Catherine Arnett, Marisa Hudspeth, Brendan O'Connor
While tokenization is a key step in language modeling, with effects on model training and performance, it remains unclear how to effectively evaluate tokenizer quality. One propose…
BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues
Prashant Jayannavar, Liliang Ren, Marisa Hudspeth +6
Developing interactive agents that can understand language, perceive their surroundings, and act within the physical world is a long-standing goal of AI research. The Minecraft Col…
Latin Treebanks in Review: An Evaluation of Morphological Tagging Across Time
Marisa Hudspeth, Brendan O'Connor, Laure Thompson
Existing Latin treebanks draw from Latin's long written tradition, spanning 17 centuries and a variety of cultures. Recent efforts have begun to harmonize these treebanks' annotati…
Finding Cortical Subregions Regarding the Dorsal Language Pathway Based on the Structural Connectivity
Young-Eun Hwang, Young-Bo Kim, Young-Don Son
Although the language-related fiber pathways in the human brain, such as superior longitudinal fasciculus (SLF) and arcuate fasciculus (AF), are already well-known, understanding m…