6 papers · 1 filter
Multilingual Multimodal Software Developer for Code Generation
Linzheng Chai, Jian Yang, Shukai Liu +12
The rapid advancement of Large Language Models (LLMs) has significantly improved code generation, yet most models remain text-only, neglecting crucial visual aids like diagrams and…
MdEval: Massively Multilingual Code Debugging
Shukai Liu, Linzheng Chai, Jian Yang +15
Code large language models (LLMs) have made significant progress in code debugging by directly generating the correct code based on the buggy code snippet. Programming benchmarks,…
SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models
Xianfu Cheng, Wei Zhang, Shiwei Zhang +16
The increasing application of multi-modal large language models (MLLMs) across various sectors have spotlighted the essence of their output reliability and accuracy, particularly t…
ExecRepoBench: Multi-level Executable Code Completion Evaluation
Jian Yang, Jiajun Zhang, Jiaxi Yang +9
Code completion has become an essential tool for daily software development. Existing evaluation benchmarks often employ static methods that do not fully capture the dynamic nature…
Evaluating and Aligning CodeLLMs on Human Preference
Jian Yang, Jiaxi Yang, Ke Jin +7
Code large language models (codeLLMs) have made significant strides in code generation. Most previous code-related benchmarks, which consist of various programming exercises along…
M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation
Jiaheng Liu, Ken Deng, Congnan Liu +13
Repository-level code completion has drawn great attention in software engineering, and several benchmark datasets have been introduced. However, existing repository-level code com…