Beyond Productivity: Rethinking the Impact of Creativity Support Tools
arXiv:2505.01601 · doi:10.1145/3698061.3726924
Abstract
Creativity Support Tools (CSTs) are widely used across diverse creative domains, with generative AI recently increasing the abilities of CSTs. To better understand how the success of CSTs is determined in the literature, we conducted a review of outcome measures used in CST evaluations. Drawing from (n=173) CST evaluations in the ACM Digital Library, we identified the metrics commonly employed to assess user interactions with CSTs. Our findings reveal prevailing trends in current evaluation practices, while exposing underexplored measures that could broaden the scope of future research. Based on these results, we argue for a more holistic approach to evaluating CSTs, encouraging the HCI community to consider not only user experience and the quality of the generated output, but also user-centric aspects such as self-reflection and well-being as critical dimensions of assessment. We also highlight a need for validated measures specifically suited to the evaluation of generative AI in CSTs.
In ACM Creativity and Cognition (C&C '25), June 23-25, 2025; 15 pages; 2 Figures; 3 Tables
References in corpus (26)
- Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems
- The Effects of Generative AI on Design Fixation and Divergent Thinking
- Homogenization Effects of Large Language Models on Human Creative Ideation
- If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills
- UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library
- Mathemyths: Leveraging Large Language Models to Teach Mathematical Language through Child-AI Co-Creative Storytelling
- Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts
- The Development and Validation of the Technology-Supported Reflection Inventory
- GRIDS: Interactive Layout Design with Integer Programming
- ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12
- Sketchar: Supporting Character Design and Illustration Prototyping Using Generative AI
- CharacterChat: Supporting the Creation of Fictional Characters through Conversation and Progressive Manifestation with a Chatbot
- TypeDance: Creating Semantic Typographic Logos from Image through Personalized Generation
- Spinneret: Aiding Creative Ideation through Non-Obvious Concept Associations
- Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideation
- pARam: Leveraging Parametric Design in Extended Reality to Support the Personalization of Artifacts for Personal Fabrication
- Chatbots for Data Collection in Surveys: A Comparison of Four Theory-Based Interview Probes
- AI-Assisted Causal Pathway Diagram for Human-Centered Design
- Portrayal: Leveraging NLP and Visualization for Analyzing Fictional Characters
- Cine-AI: Generating Video Game Cutscenes in the Style of Human Directors
- AdapTics: A Toolkit for Creative Design and Integration of Real-Time Adaptive Mid-Air Ultrasound Tactons
- EditScribe: Non-Visual Image Editing with Natural Language Verification Loops
- MIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on Videos
- Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review
- From Interaction to Attitude: Exploring the Impact of Human-AI Cooperation on Mental Illness Stigma
- The Benefits of Prosociality towards AI Agents: Examining the Effects of Helping AI Agents on Human Well-Being