Memorize or Generalize? Evaluating LLM Code Generation with Code Rewriting

Lizhe Zhang, Wentao Chen, Li Zhong +3

Large language models (LLMs) have recently demonstrated exceptional code generation capabilities. However, there is a growing debate whether LLMs are mostly doing memorization (i.e…