No Evidence for LLMs Being Useful in Problem Reframing

Abstract

Problem reframing is a designerly activity wherein alternative perspectives are created to recast what a stated design problem is about. Generating alternative problem frames is challenging because it requires devising novel and useful perspectives that fit the given problem context. Large language models (LLMs) could assist this activity via their generative capability. However, it is not clear whether they can help designers produce high-quality frames. Therefore, we asked if there are benefits to working with LLMs. To this end, we compared three ways of using LLMs (N=280): 1) free-form, 2) direct generation, and 3) a structured approach informed by a theory of reframing. We found that using LLMs does not help improve the quality of problem frames. In fact, it increases the competence gap between experienced and inexperienced designers. Also, inexperienced ones perceived lower agency when working with LLMs. We conclude that there is no benefit to using LLMs in problem reframing and discuss possible factors for this lack of effect.

14 pages, 10 figures, 2 tables, Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems

References in corpus (9)

• Generative artificial intelligence enhances creativity but reduces the diversity of novel content
• The Effects of Generative AI on Design Fixation and Divergent Thinking
• Homogenization Effects of Large Language Models on Human Creative Ideation
• Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models
• Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation
• Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniques
• Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation
• An Evaluation on Large Language Model Outputs: Discourse and Memorization
• Investigating Interaction Modes and User Agency in Human-LLM Collaboration for Domain-Specific Data Analysis