84 citations · 313 across the 28 of their papers we have counts for
31 papers
Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations
Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi +2
As large language models (LLMs) gain popularity among speakers of diverse languages, we believe that it is crucial to benchmark them to better understand model behaviors, failures,…
Causal schema induction for knowledge discovery
Michael Regan, Jena D. Hwang, Keisuke Sakaguchi +1
Making sense of familiar yet new situations typically involves making generalizations about causal schemas, stories that help humans reason about event sequences. Reasoning about e…
Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction
Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa +2
GPT-3 and GPT-4 models are powerful, achieving high performance on a variety of Natural Language Processing tasks. However, there is a relative lack of detailed published analysis…
Empirical Investigation of Neural Symbolic Reasoning Strategies
Yoichi Aoki, Keito Kudo, Tatsuki Kuribayashi +4
Neural reasoning accuracy improves when generating intermediate reasoning steps. However, the source of this improvement is yet unclear. Here, we investigate and factorize the bene…
Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?
Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi +4
Compositionality is a pivotal property of symbolic reasoning. However, how well recent neural models capture compositionality remains underexplored in the symbolic reasoning tasks.…
I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation
Chandra Bhagavatula, Jena D. Hwang, Doug Downey +7
Commonsense capabilities of pre-trained language models dramatically improve with scale, leading many to believe that scale is the only winning recipe. But is it? Here, we investig…
RealTime QA: What's the Answer Right Now?
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi +7
We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). REALTIME QA inquire…
Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond
Masato Mita, Keisuke Sakaguchi, Masato Hagiwara +3
Natural language processing technology has rapidly improved automated grammatical error correction tasks, and the community begins to explore document-level revision as one of the…
Twist Decoding: Diverse Generators Guide Each Other
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras +5
Many language generation models are now available for a wide range of generation tasks, including machine translation and summarization. Combining such diverse models may lead to f…
ELQA: A Corpus of Metalinguistic Questions and Answers about English
Shabnam Behzad, Keisuke Sakaguchi, Nathan Schneider +1
We present ELQA, a corpus of questions and answers in and about the English language. Collected from two online forums, the >70k questions (from English learners and others) cover…
A Call for Clarity in Beam Search: How It Works and When It Stops
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras +3
Text generation with beam search has proven successful in a wide range of applications. We point out that, though largely overlooked in the literature, the commonly-used implementa…
Interscript: A dataset for interactive learning of scripts through error feedback
Niket Tandon, Aman Madaan, Peter Clark +2
How can an end-user provide feedback if a deployed structured prediction model generates inconsistent output, ignoring the structural complexity of human language? This is an emerg…
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras +5
Natural language processing researchers have identified limitations of evaluation methodology for generation tasks, with new questions raised about the validity of automatic metric…
Transparent Human Evaluation for Image Captioning
Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan +4
We establish THumB, a rubric-based human evaluation protocol for image captioning models. Our scoring rubrics and their definitions are carefully developed based on machine- and hu…
Can Machines Learn Morality? The Delphi Experiment
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula +12
As AI systems become increasingly powerful and pervasive, there are growing concerns about machines' morality or a lack thereof. Yet, teaching morality to machines is a formidable…
Improving Neural Model Performance through Natural Language Feedback on Their Explanations
Aman Madaan, Niket Tandon, Dheeraj Rajagopal +4
A class of explainable NLP models for reasoning tasks support their decisions by generating free-form or structured explanations, but what happens when these supporting structures…
proScript: Partially Ordered Scripts Generation via Pre-trained Language Models
Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras +3
Scripts - standardized event sequences describing typical everyday activities - have been shown to help understand narratives by providing expectations, resolving ambiguity, and fi…
GrammarTagger: A Multilingual, Minimally-Supervised Grammar Profiler for Language Education
Masato Hagiwara, Joshua Tanner, Keisuke Sakaguchi
We present GrammarTagger, an open-source grammar profiler which, given an input text, identifies grammatical features useful for language education. The model architecture enables…
A Dataset for Tracking Entities in Open Domain Procedural Text
Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi Mishra +5
We present the first dataset for tracking state changes in procedural text from arbitrary domains by using an unrestricted (open) vocabulary. For example, in a text describing fog…
COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras +4
Recent years have brought about a renewed interest in commonsense representation and reasoning in the field of natural language understanding. The development of new commonsense kn…
The Universal Decompositional Semantics Dataset and Decomp Toolkit
Aaron Steven White, Elias Stengel-Eskin, Siddharth Vashishtha +9
We present the Universal Decompositional Semantics (UDS) dataset (v1.0), which is bundled with the Decomp toolkit (v0.1). UDS1.0 unifies five high-quality, decompositional semantic…
WIQA: A dataset for "What if..." reasoning over procedural text
Niket Tandon, Bhavana Dalvi Mishra, Keisuke Sakaguchi +2
We introduce WIQA, the first large-scale dataset of "What if..." questions over procedural text. WIQA contains three parts: a collection of paragraphs each describing a process, e.…
Uncertain Natural Language Inference
Tongfei Chen, Zhengping Jiang, Adam Poliak +2
We introduce Uncertain Natural Language Inference (UNLI), a refinement of Natural Language Inference (NLI) that shifts away from categorical labels, targeting instead the direct pr…
Abductive Commonsense Reasoning
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya +6
Abductive reasoning is inference to the most plausible explanation. For example, if Jenny finds her house in a mess when she returns from work, and remembers that she left a window…