7 papers
DVD: A Robust Method for Detecting Variant Contamination in Large Language Model Evaluation
Renzhao Liang, Jingru Chen, Bo Jia +7
Evaluating large language models (LLMs) is increasingly confounded by \emph{variant contamination}: the training corpus contains semantically equivalent yet lexically or syntactica…
Multilingual Multimodal Software Developer for Code Generation
Linzheng Chai, Jian Yang, Shukai Liu +12
The rapid advancement of Large Language Models (LLMs) has significantly improved code generation, yet most models remain text-only, neglecting crucial visual aids like diagrams and…
MdEval: Massively Multilingual Code Debugging
Shukai Liu, Linzheng Chai, Jian Yang +15
Code large language models (LLMs) have made significant progress in code debugging by directly generating the correct code based on the buggy code snippet. Programming benchmarks,…
SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models
Xianfu Cheng, Wei Zhang, Shiwei Zhang +16
The increasing application of multi-modal large language models (MLLMs) across various sectors have spotlighted the essence of their output reliability and accuracy, particularly t…
ExecRepoBench: Multi-level Executable Code Completion Evaluation
Jian Yang, Jiajun Zhang, Jiaxi Yang +9
Code completion has become an essential tool for daily software development. Existing evaluation benchmarks often employ static methods that do not fully capture the dynamic nature…
Evaluating and Aligning CodeLLMs on Human Preference
Jian Yang, Jiaxi Yang, Ke Jin +7
Code large language models (codeLLMs) have made significant strides in code generation. Most previous code-related benchmarks, which consist of various programming exercises along…