80 citations · 297 across the 46 of their papers we have counts for
13 papers · 1 filter
When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks
Hao Peng, Xiaozhi Wang, Jianhui Chen +8
In-context learning (ICL) has become the default method for using large language models (LLMs), making the exploration of its limitations and understanding the underlying causes cr…
MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation
Xiaozhi Wang, Hao Peng, Yong Guan +9
Understanding events in texts is a core objective of natural language understanding, which requires detecting event occurrences, extracting event arguments, and analyzing inter-eve…
Mastering the Task of Open Information Extraction with Large Language Models and Consistent Reasoning Environment
Ji Qi, Kaixuan Ji, Xiaozhi Wang +5
Open Information Extraction (OIE) aims to extract objective structured knowledge from natural texts, which has attracted growing attention to build dedicated models with human expe…
OmniEvent: A Comprehensive, Fair, and Easy-to-Use Toolkit for Event Understanding
Hao Peng, Xiaozhi Wang, Feng Yao +5
Event understanding aims at understanding the content and relationship of events within texts, which covers multiple complicated information extraction tasks: event detection, even…
The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation
Hao Peng, Xiaozhi Wang, Feng Yao +5
Event extraction (EE) is a crucial task aiming at extracting events from texts, which includes two subtasks: event detection (ED) and event argument extraction (EAE). In this paper…
KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Jifan Yu, Xiaozhi Wang, Shangqing Tu +32
The unprecedented performance of large language models (LLMs) necessitates improvements in evaluations. Rather than merely exploring the breadth of LLM abilities, we believe meticu…
Benchmarking Foundation Models with Language-Model-as-an-Examiner
Yushi Bai, Jiahao Ying, Yixin Cao +10
Numerous benchmarks have been established to assess the performance of foundation models on open-ended question answering, which serves as a comprehensive test of a model's ability…
Emergent Modularity in Pre-trained Transformers
Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin +7
This work examines the presence of modularity in pre-trained Transformers, a feature commonly found in human brains and thought to be vital for general intelligence. In analogy to…
Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction
Ji Qi, Chuchun Zhang, Xiaozhi Wang +8
The robustness to distribution changes ensures that NLP models can be successfully applied in the realistic world, especially for information extraction tasks. However, most prior…
ChatLog: Carefully Evaluating the Evolution of ChatGPT Across Time
Shangqing Tu, Chunyang Li, Jifan Yu +3
ChatGPT has achieved great success and can be considered to have acquired an infrastructural status. There are abundant works for evaluating ChatGPT on benchmarks. However, existin…
GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generation
Ji Qi, Jifan Yu, Teng Tu +12
Despite the recent emergence of video captioning models, how to generate vivid, fine-grained video descriptions based on the background knowledge (i.e., long and informative commen…
Language-Specific Representation of Emotion-Concept Knowledge Causally Supports Emotion Inference
Ming Li, Yusheng Su, Hsiu-Yuan Huang +9
Humans no doubt use language to communicate about their emotional experiences, but does language in turn help humans understand emotions, or is language just a vehicle of communica…
READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises
Chenglei Si, Zhengyan Zhang, Yingfa Chen +3
For many real-world applications, the user-generated inputs usually contain various noises due to speech recognition errors caused by linguistic variations1 or typographical errors…