9 papers
When Drawing Is Not Enough: Exploring Spontaneous Speech with Sketch for Intent Alignment in Multimodal LLMs
Weiyan Shi, Dorien Herremans, Kenny Tsu Wei Choo
Early-stage design ideation often relies on rough sketches created under time pressure, leaving much of the designer's intent implicit. In practice, designers frequently speak whil…
More Than 1v1: Human-AI Alignment in Early Developmental Communities with Multimodal LLMs
Weiyan Shi, Kenny Tsu Wei Choo
In early developmental contexts, particularly in parent-child interaction analysis, alignment involves families and professionals such as speech-language pathologists (SLPs) who in…
Towards Aligning Multimodal LLMs with Human Experts: A Focus on Parent-Child Interaction
Weiyan Shi, Kenny Tsu Wei Choo
While multimodal large language models (MLLMs) are increasingly applied in human-centred AI systems, their ability to understand complex social interactions remains uncertain. We p…
A Taxonomy of Human--MLLM Interaction in Early-Stage Sketch-Based Design Ideation
Weiyan Shi, Kenny Tsu Wei Choo
As multimodal large language models (MLLMs) are increasingly integrated into early-stage design tools, it is important to understand how designers collaborate with AI during ideati…
Human-AI Alignment of Multimodal Large Language Models with Speech-Language Pathologists in Parent-Child Interactions
Weiyan Shi, Kenny Tsu Wei Choo
Joint attention is a critical marker of early social-communicative development, yet remains difficult for caregivers to assess without expert guidance. In this work, we explore how…
TalkSketch: Multimodal Generative AI for Real-time Sketch Ideation with Speech
Weiyan Shi, Sunaya Upadhyay, Geraldine Quek +1
Sketching is a widely used medium for generating and exploring early-stage design concepts. While generative AI (GenAI) chatbots are increasingly used for idea generation, designer…