Assessing and Understanding Creativity in Large Language Models

Abstract

In the field of natural language processing, the rapid development of large language model (LLM) has attracted more and more attention. LLMs have shown a high level of creativity in various tasks, but the methods for assessing such creativity are inadequate. The assessment of LLM creativity needs to consider differences from humans, requiring multi-dimensional measurement while balancing accuracy and efficiency. This paper aims to establish an efficient framework for assessing the level of creativity in LLMs. By adapting the modified Torrance Tests of Creative Thinking, the research evaluates the creative performance of various LLMs across 7 tasks, emphasizing 4 criteria including Fluency, Flexibility, Originality, and Elaboration. In this context, we develop a comprehensive dataset of 700 questions for testing and an LLM-based evaluation method. In addition, this study presents a novel analysis of LLMs' responses to diverse prompts and role-play situations. We found that the creativity of LLMs primarily falls short in originality, while excelling in elaboration. Besides, the use of prompts and the role-play settings of the model significantly influence creativity. Additionally, the experimental results also indicate that collaboration among multiple LLMs can enhance originality. Notably, our findings reveal a consensus between human evaluations and LLMs regarding the personality traits that influence creativity. The findings underscore the significant impact of LLM design on creativity and bridges artificial intelligence and human creativity, offering insights into LLMs' creativity and potential applications.

References in corpus (13)

• Diffusion Models in Vision: A Survey
• A Survey on Large Language Model based Autonomous Agents
• Homogenization Effects of Large Language Models on Human Creative Ideation
• On the Creativity of Large Language Models
• Putting GPT-3's Creativity to the (Alternative Uses) Test
• ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
• PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• Benchmarking Foundation Models with Language-Model-as-an-Examiner
• On the Tool Manipulation Capability of Open-source Large Language Models
• CARE-MI: Chinese Benchmark for Misinformation Evaluation in Maternity and Infant Care
• Creative Data Generation: A Review Focusing on Text and Poetry
• Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset

Cited by in corpus (2)

• Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons
• Computational Hermeneutics: Evaluating generative AI as a cultural technology