5 papers
Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis
Hong Nguyen, Sean Foley, Christina Hagedorn +4
Real-time MRI (rtMRI) captures the dynamics of the entire vocal tract during speech, but labeled data are scarce and the modality - single-slice, grayscale, low-resolution - differ…
Interpretable Modeling of Articulatory Temporal Dynamics from real-time MRI for Phoneme Recognition
Jay Park, Hong Nguyen, Sean Foley +4
Real-time Magnetic Resonance Imaging (rtMRI) visualizes vocal tract action, offering a comprehensive window into speech articulation. However, its signals are high dimensional and…
MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows
Hong Nguyen, Dung Tran, Hieu Hoang +2
Many motion-centric video analysis tasks, such as atomic actions, detecting atypical motor behavior in individuals with autism, or analyzing articulatory motion in real-time MRI of…
Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition
Sean Foley, Hong Nguyen, Jihwan Lee +4
Although many previous studies have carried out multimodal learning with real-time MRI data that captures the audio-visual kinematics of the vocal tract during speech, these studie…
Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech
Hong Nguyen, Sean Foley, Kevin Huang +3
Understanding speech production both visually and kinematically can inform second language learning system designs, as well as the creation of speaking characters in video games an…