29 citations · 66 across the 18 of their papers we have counts for
21 papers
Native Action-Prior Learning from Videos for World Action Models
Zhaochong An, Fei Zhang, Menglin Jia +10
World action models integrate future visual dynamics with robot action prediction, but their scalability remains limited by the need for action-annotated robot trajectories. Observ…
Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation
Zhiheng Liu, Weiming Ren, Xiaoke Huang +12
Unified multimodal models typically rely on pretrained vision encoders and use separate visual representations for understanding and generation, creating misalignment between the t…
TransText: Alpha-as-RGB Representation for Transparent Text Animation
Fei Zhang, Zijian Zhou, Bohao Tang +9
We introduce the first method, to the best of our knowledge, for adapting image-to-video models to layer-aware text (glyph) animation, a capability critical for practical dynamic v…
VecGlypher: Unified Vector Glyph Generation with Language Models
Xiaoke Huang, Bhavul Gauri, Kam Woh Ng +12
Vector glyphs are the atomic units of digital typography, yet most learning-based pipelines still depend on carefully curated exemplar sheets and raster-to-vector postprocessing, w…
HiStream: Efficient High-Resolution Video Generation via Redundancy-Eliminated Streaming
Haonan Qiu, Shikun Liu, Zijian Zhou +10
High-resolution video generation, while crucial for digital media and film, is computationally bottlenecked by the quadratic complexity of diffusion models, making practical infere…
OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory
Zhaochong An, Menglin Jia, Haonan Qiu +12
Storytelling in real-world videos often unfolds through multiple shots -- discontinuous yet semantically connected clips that together convey a coherent narrative. However, existin…
Scaling Zero-Shot Reference-to-Video Generation
Zijian Zhou, Shikun Liu, Haozhe Liu +14
Reference-to-video (R2V) generation aims to synthesize videos that align with a text prompt while preserving the subject identity from reference images. However, current R2V method…
TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models
Zhiheng Liu, Weiming Ren, Haozhe Liu +22
Unified multimodal models (UMMs) aim to jointly perform multimodal understanding and generation within a single framework. We present TUNA, a native UMM that builds a unified conti…
Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
Haozhe Liu, Ding Liu, Mingchen Zhuge +16
We introduce MoS (Mixture of States), a novel fusion paradigm for multimodal diffusion models that merges modalities using flexible, state-based interactions. The core of MoS is a…
Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset
Vasu Agrawal, Akinniyi Akinyemi, Kathryn Alvero +81
Human communication involves a complex interplay of verbal and nonverbal signals, essential for conveying meaning and achieving interpersonal goals. To develop socially intelligent…
Learning Flow Fields in Attention for Controllable Person Image Generation
Zijian Zhou, Shikun Liu, Xiao Han +11
Controllable person image generation aims to generate a person image conditioned on reference images, allowing precise control over the person's appearance or pose. However, prior…
Adaptive Caching for Faster Video Generation with Diffusion Transformers
Kumara Kahatapitiya, Haozhe Liu, Sen He +5
Generating temporally-consistent high-fidelity videos can be computationally expensive, especially over longer temporal spans. More-recent Diffusion Transformers (DiTs) -- despite…
MarDini: Masked Autoregressive Diffusion for Video Generation at Scale
Haozhe Liu, Shikun Liu, Zijian Zhou +12
We introduce MarDini, a new family of video diffusion models that integrate the advantages of masked auto-regression (MAR) into a unified diffusion model (DM) framework. Here, MAR…
Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object Structure via HyperNetworks
Christian Simon, Sen He, Juan-Manuel Perez-Rua +3
Solving image-to-3D from a single view is an ill-posed problem, and current neural reconstruction methods addressing it through diffusion models still rely on scene-specific optimi…
GenTron: Diffusion Transformers for Image and Video Generation
Shoufa Chen, Mengmeng Xu, Jiawei Ren +7
In this study, we explore Transformer-based diffusion models for image and video generation. Despite the dominance of Transformer architectures in various fields due to their flexi…
FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing
Yuren Cong, Mengmeng Xu, Christian Simon +7
Text-to-video editing aims to edit the visual appearance of a source video conditional on textual prompts. A major challenge in this task is to ensure that all frames in the edited…
Learning Garment DensePose for Robust Warping in Virtual Try-On
Aiyu Cui, Sen He, Tao Xiang +1
Virtual try-on, i.e making people virtually try new garments, is an active research area in computer vision with great commercial applications. Current virtual try-on methods usual…
Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
Michał Stypułkowski, Konstantinos Vougioukas, Sen He +3
Talking face generation has historically struggled to produce head movements and natural facial expressions without guidance from additional reference videos. Recent developments i…
Prediction Calibration for Generalized Few-shot Semantic Segmentation
Zhihe Lu, Sen He, Da Li +2
Generalized Few-shot Semantic Segmentation (GFSS) aims to segment each image pixel into either base classes with abundant training examples or novel classes with only a handful of…
Hybrid Graph Neural Networks for Few-Shot Learning
Tianyuan Yu, Sen He, Yi-Zhe Song +1
Graph neural networks (GNNs) have been used to tackle the few-shot learning (FSL) problem and shown great potentials under the transductive setting. However under the inductive set…
Deep saliency: What is learnt by a deep network about saliency?
Sen He, Nicolas Pugeault
Deep convolutional neural networks have achieved impressive performance on a broad range of problems, beating prior art on established benchmarks, but it often remains unclear what…