From the 1 of 12 linked papers with an AI index.
12 papers
Linear representations of grammaticality in neural language models
Jane Li, Najoung Kim
The paper investigates whether neural language models encode grammaticality in their internal sentence representations, using mass‑mean probing to show a systematic separation betw…
Do Language Models Track Entities Across State Changes?
Zilu Tang, Qiao Zhao, Gabriel Franco +4
Entity tracking (ET), the ability to keep track of states, is a fundamental skill that underlies complex reasoning. An increasing amount of work investigates how transformer langua…
RExBench: Can coding agents autonomously implement AI research extensions?
Nicholas Edwards, Yukyung Lee, Yujun Audrey Mao +3
Agents based on Large Language Models (LLMs) have shown promise for performing sophisticated software engineering tasks autonomously. In addition, there has been progress towards d…
Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues
Eunsu Kim, Junyeong Park, Juhyun Oh +5
As LLMs are increasingly deployed in real-world interactions, their social reasoning in interpersonal communication becomes critical. To explore their capabilities, we introduce SC…
A systematic framework for generating novel experimental hypotheses from language models
Kanishka Misra, Najoung Kim
Neural language models (LMs) have been shown to capture complex linguistic patterns, yet their utility in understanding human language and more broadly, human cognition, remains de…
Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity
Arkadiy Saakyan, Najoung Kim, Smaranda Muresan +1
N-gram novelty is widely used to evaluate language models' ability to generate text outside of their training data. More recently, it has also been adopted as a metric for measurin…