MemoVis: A GenAI-Powered Tool for Creating Companion Reference Images for 3D Design Feedback
arXiv:2409.06082 · doi:10.1145/3694681
Abstract
Providing asynchronous feedback is a critical step in the 3D design workflow. A common approach to providing feedback is to pair textual comments with companion reference images, which helps illustrate the gist of text. Ideally, feedback providers should possess 3D and image editing skills to create reference images that can effectively describe what they have in mind. However, they often lack such skills, so they have to resort to sketches or online images which might not match well with the current 3D design. To address this, we introduce MemoVis, a text editor interface that assists feedback providers in creating reference images with generative AI driven by the feedback comments. First, a novel real-time viewpoint suggestion feature, based on a vision-language foundation model, helps feedback providers anchor a comment with a camera viewpoint. Second, given a camera viewpoint, we introduce three types of image modifiers, based on pre-trained 2D generative models, to turn a text comment into an updated version of the 3D scene from that viewpoint. We conducted a within-subjects study with feedback providers, demonstrating the effectiveness of MemoVis. The quality and explicitness of the companion images were evaluated by another eight participants with prior 3D design experience.
In the Journal of ACM Transactions on Computer-Human Interaction
References in corpus (30)
- Generative Adversarial Networks
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- High-Resolution Image Synthesis with Latent Diffusion Models
- Segment Anything
- Fine-Tuning Language Models from Human Preferences
- eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
- Adding Conditional Control to Text-to-Image Diffusion Models
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
- Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works
- PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
- PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
- How does HCI Understand Human Autonomy and Agency?
- Sequential Gallery for Interactive Visual Design Optimization
- Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models
- Levels of AGI for Operationalizing Progress on the Path to AGI
- Optimizing Prompts for Text-to-Image Generation
- GANzilla: User-Driven Direction Discovery in Generative Adversarial Networks
- Recognize Anything: A Strong Image Tagging Model
- GANravel: User-Driven Direction Disentanglement in Generative Adversarial Networks
- Next Steps for Human-Centered Generative AI: A Technical Perspective
- BlendScape: Enabling End-User Customization of Video-Conferencing Environments through Generative AI
- Tag2Text: Guiding Vision-Language Model via Image Tagging
- InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructions
- Improving Text-to-Image Consistency via Automatic Prompt Optimization
- GenQuery: Supporting Expressive Visual Search with Generative Models
- InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes
- Efficient Quantization Strategies for Latent Diffusion Models
- LooseControl: Lifting ControlNet for Generalized Depth Conditioning
- Paparazzi: A Deep Dive into the Capabilities of Language and Vision Models for Grounding Viewpoint Descriptions