158 citations · 646 across the 34 of their papers we have counts for
7 papers · 1 filter
Jointly Learning Visual and Auditory Speech Representations from Raw Data
Alexandros Haliassos, Pingchuan Ma, Rodrigo Mira +2
We present RAVEn, a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our pre-training objective involves encoding masked inputs, an…
Streaming Audio-Visual Speech Recognition with Alignment Regularization
Pingchuan Ma, Niko Moritz, Stavros Petridis +2
In this work, we propose a streaming AV-ASR system based on a hybrid connectionist temporal classification (CTC)/attention neural network architecture. The audio and the visual enc…
Training Strategies for Improved Lip-reading
Pingchuan Ma, Yujiang Wang, Stavros Petridis +2
Several training strategies and temporal models have been recently proposed for isolated word lip-reading in a series of independent works. However, the potential of combining the…
RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation
Pingchuan Ma, Tao Du, Joshua B. Tenenbaum +2
This work considers identifying parameters characterizing a physical system's dynamic motion directly from a video whose rendering configurations are inaccessible. Existing solutio…
Fast Aquatic Swimmer Optimization with Differentiable Projective Dynamics and Neural Network Hydrodynamic Models
Elvis Nava, John Z. Zhang, Mike Y. Michelis +5
Aquatic locomotion is a classic fluid-structure interaction (FSI) problem of interest to biologists and engineers. Solving the fully coupled FSI equations for incompressible Navier…
Self-supervised Video-centralised Transformer for Video Face Clustering
Yujiang Wang, Mingzhi Dong, Jie Shen +5
This paper presents a novel method for face clustering in videos using a video-centralised transformer. Previous works often employed contrastive learning to learn frame-level repr…