8 papers
The Plot Thins: Uniformity and Linearity in Literary Summaries
Rebecca M. M. Hicke, Sil Hamilton, David Mimno +1
Works of literature are complicated; they balance plot, suspense, surprise, and artistic expression. Summaries of literature prioritize plot, and therefore may deviate from their s…
Elias in the Lighthouse, Again? Diagnosing Low Diversity in LLM Stories
Sil Hamilton, David Mimno
LLM-generated stories are a popular use case, but they show very low variability. We sample 20,000 total stories from four current models using five prompts. We find that 11 words…
Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries
Rebecca M. M. Hicke, Sil Hamilton, David Mimno +1
Although LLM context lengths have grown, there is evidence that their ability to integrate information across long-form texts has not kept pace. We evaluate one such understanding…
NarraBench: A Comprehensive Framework for Narrative Benchmarking
Sil Hamilton, Matthew Wilkens, Andrew Piper
We present NarraBench, a theory-informed taxonomy of narrative-understanding tasks, as well as an associated survey of 78 existing benchmarks in the area. We find significant need…
Lost in Space: Finding the Right Tokens for Structured Output
Sil Hamilton, David Mimno
General-purpose language models are trained to produce varied natural language outputs, but for some tasks, like annotation or classification, we need more specific output formats.…
Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels
Sil Hamilton, Rebecca M. M. Hicke, Matthew Wilkens +1
Although the context length of large language models (LLMs) has increased to millions of tokens, evaluating their effectiveness beyond needle-in-a-haystack approaches has proven di…