MaterialVellum: Steering Generative AI by Leveraging Expressive Material Affordances to Form Design Constraints

Abstract

We present MaterialVellum, an interaction approach for steering generative AI through material-inspired manipulation of design constraints. In urban design, for instance, zoning bounds can be defined by the stretch of elastic bands, building density redistributed by molding clay, and functional relationships organized by the pull of magnets. MaterialVellum departs from current interfaces that rely on natural language prompts, and instead leverages users' existing knowledge of physical materials to represent and manipulate design constraints. We introduce a framework that maps material affordances---such as elasticity, malleability, or attraction---to spatial, relational, and proportional constraints that underpin design. These mappings are operationalized in user interfaces through familiar direct manipulation multi-touch gestures, including stretching, sculpting, and clustering, whose effects feed as control parameters into generative AI pipelines. We demonstrate this approach through technology probes, including an urban design, a flower arrangement design, and a graphical design scenario.A qualitative study (N=10) shows that material-based interactions improve accessibility of constraint expression, support exploratory workflows, and lower barriers for non-experts, while effectiveness depends on alignment with users' mental models.

References in corpus (5)

• DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models
• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
• Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints
• AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools
• Texterial: A Text-as-Material Interaction Paradigm for LLM-Mediated Writing