8 papers
XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics
Jingxuan Liu, Zhi Qu, Jin Tei +3
Automatic evaluation metrics are essential for building multilingual translation systems. The common practice of evaluating these systems is averaging metric scores across language…
Judge Like Human Examiners: A Weighted Importance Multi-Point Evaluation Framework for Generative Tasks with Long-form Answers
Guoxin Yu, Chulun Zhou, Lemao Liu +7
Evaluating the quality of model responses remains challenging in generative tasks with long-form answers, as the expected answers usually contain multiple semantically distinct yet…
A Decomposition Perspective to Long-context Reasoning for LLMs
Yanling Xiao, Huaibing Xie, Guoliang Zhao +8
Long-context reasoning is essential for complex real-world applications, yet remains a significant challenge for Large Language Models (LLMs). Despite the rapid evolution in long-c…
DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models
Tsz Ting Chung, Lemao Liu, Mo Yu +1
Logic reasoning in natural language has been recognized as an important measure of human intelligence for Large Language Models (LLMs). Popular benchmarks may entangle multiple rea…
SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension
Junjie Wu, Jiangnan Li, Yuqing Li +6
Retrieval-augmented generation (RAG) over long documents typically involves splitting the text into smaller chunks, which serve as the basic units for retrieval. However, due to de…
Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Task
Junjie Wu, Mo Yu, Lemao Liu +2
While LLMs have exhibited strong performance on various NLP tasks, it is noteworthy that most of these tasks rely on utilizing the vast amount of knowledge encoded in LLMs' paramet…
ReFreeKV: Towards Threshold-Free KV Cache Compression
Xuanfan Ni, Liyan Xu, Chenyang Lyu +6
To reduce memory consumption during LLM inference, a handful of methods have been proposed for KV cache pruning. While these techniques can accomplish lossless memory reduction on…
Large Language Models Can Self-Improve in Long-context Reasoning
Siheng Li, Cheng Yang, Zesen Cheng +4
Large language models (LLMs) have achieved substantial progress in processing long contexts but still struggle with long-context reasoning. Existing approaches typically involve fi…