5 papers
NarraBench: A Comprehensive Framework for Narrative Benchmarking
Sil Hamilton, Matthew Wilkens, Andrew Piper
We present NarraBench, a theory-informed taxonomy of narrative-understanding tasks, as well as an associated survey of 78 existing benchmarks in the area. We find significant need…
Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels
Sil Hamilton, Rebecca M. M. Hicke, Matthew Wilkens +1
Although the context length of large language models (LLMs) has increased to millions of tokens, evaluating their effectiveness beyond needle-in-a-haystack approaches has proven di…
The Zero Body Problem: Probing LLM Use of Sensory Language
Rebecca M. M. Hicke, Sil Hamilton, David Mimno
Sensory language expresses embodied experiences ranging from taste and sound to excitement and stomachache. This language is of interest to scholars from a wide range of domains in…
A City of Millions: Mapping Literary Social Networks At Scale
Sil Hamilton, Rebecca M. M. Hicke, David Mimno +1
We release 70,509 high-quality social networks extracted from multilingual fiction and nonfiction narratives. We additionally provide metadata for 30,000 of these texts (73\%…
Lost in Space: Finding the Right Tokens for Structured Output
Sil Hamilton, David Mimno
General-purpose language models are trained to produce varied natural language outputs, but for some tasks, like annotation or classification, we need more specific output formats.…