3 citations · 6 across the 6 of their papers we have counts for
7 papers
World of Workflows: A Benchmark for Bringing World Models to Enterprise Systems
Lakshya Gupta, Litao Li, Yizhe Liu +5
Frontier large language models (LLMs) excel as autonomous agents in many domains, yet they remain untested in complex enterprise systems where hidden workflows create cascading eff…
SCOPE: Language Models as One-Time Teacher for Hierarchical Planning in Text Environments
Haoye Lu, Pavan Seshadri, Kaheer Suleman
Long-term planning in complex, text-based environments presents significant challenges due to open-ended action spaces, ambiguous observations, and sparse feedback. Recent research…
Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective
Ian Porada, Alexandra Olteanu, Kaheer Suleman +2
It is increasingly common to evaluate the same coreference resolution (CR) model on multiple datasets. Do these multi-dataset evaluations allow us to draw meaningful conclusions ab…
The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources in Natural Language Understanding Systems
Akshatha Arodi, Martin Pömsl, Kaheer Suleman +3
Many state-of-the-art natural language understanding (NLU) models are based on pretrained neural language models. These models often make inferences using information from multiple…
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler +3
There are many ways to express similar things in text, which makes evaluating natural language generation (NLG) systems difficult. Compounding this difficulty is the need to assess…
Modeling Event Plausibility with Consistent Conceptual Abstraction
Ian Porada, Kaheer Suleman, Adam Trischler +1
Understanding natural language requires common sense, one aspect of which is the ability to discern the plausibility of events. While distributional models -- most recently pre-tra…
The Knowref Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution
Ali Emami, Paul Trichelair, Adam Trischler +3
We introduce a new benchmark for coreference resolution and NLI, Knowref, that targets common-sense understanding and world knowledge. Previous coreference resolution tasks can lar…