29 citations · 227 across the 67 of their papers we have counts for
9 papers · 1 filter
Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy
Yihong Dong, Jianha Xiao, Xue Jiang +7
The formal reasoning capabilities of LLMs are crucial for advancing automated software engineering. However, existing benchmarks for LLMs lack systematic evaluation based on comput…
Rethinking Repetition Problems of LLMs in Code Generation
Yihong Dong, Yuchen Liu, Xue Jiang +2
With the advent of neural language models, the performance of code generation has been significantly boosted. However, the problem of repetitions during the generation process cont…
EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations
Jia Li, Ge Li, Xuanming Zhang +6
How to evaluate Large Language Models (LLMs) in code generation remains an open question. Existing benchmarks have two limitations - data leakage and lack of domain-specific evalua…
Generating Equivalent Representations of Code By A Self-Reflection Approach
Jia Li, Ge Li, Lecheng Wang +2
Equivalent Representations (ERs) of code are textual representations that preserve the same semantics as the code itself, e.g., natural language comments and pseudocode. ERs play a…
DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
Jia Li, Ge Li, Yunfei Zhao +15
How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories…
EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories
Jia Li, Ge Li, Xuanming Zhang +2
How to evaluate Large Language Models (LLMs) in code generation is an open question. Existing benchmarks demonstrate poor alignment with real-world code repositories and are insuff…
Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
Yihong Dong, Xue Jiang, Huanyu Liu +4
Recent statements about the impressive capabilities of large language models (LLMs) are usually supported by evaluating on open-access benchmarks. Considering the vast size and wid…
PACE: Improving Prompt with Actor-Critic Editing for Large Language Model
Yihong Dong, Kangcheng Luo, Xue Jiang +2
Large language models (LLMs) have showcased remarkable potential across various tasks by conditioning on prompts. However, the quality of different human-written prompts leads to s…
Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively
Haojie Zhang, Ge Li, Jia Li +3
Large-scale pre-trained language models have achieved impressive results on a wide range of downstream tasks recently. However, fine-tuning an extremely large-scale pre-trained lan…