11 papers
OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence
Peigen Liu, Rui Ding, Yuren Mao +7
Large Language Model (LLM)-based Collective Intelligence (CI) presents a promising approach to overcoming the data wall and continuously boosting the capabilities of LLM agents. Ho…
MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models
Haohang Huang, Xuan Lu, Mingyi Su +9
Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space. However, existing methods and benchmarks remain…
KnowPilot: Your Knowledge-Driven Copilot for Domain Tasks
Zekun Xi, Yichen Nie, Ziyan Jiang +5
Despite the rapid advancement of generative agents, their deployment in real-world industry scenarios often encounters significant challenges due to a lack of domain-specific knowl…
StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs
Jialin Yang, Dongfu Jiang, Lipeng He +17
As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce Struct…
LightMem: Lightweight and Efficient Memory-Augmented Generation
Jizhan Fang, Xinle Deng, Haoming Xu +9
Despite their remarkable capabilities, Large Language Models (LLMs) struggle to effectively leverage historical interaction information in dynamic and complex environments. Memory…
ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders
Ofer Meshi, Krisztian Balog, Sally Goldman +5
The promise of LLM-based user simulators to improve conversational AI is hindered by a critical "realism gap," leading to systems that are optimized for simulated interactions, but…