9 papers
FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation
Guangyu Sun, Shlok Kumar Mishra, Wentao Bao +8
Traditional multimodal representation learning and generation are two stages: a contrastive or self-supervised visual encoder is trained first, followed by a separate downstream ge…
MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models
Xuanming Cui, Shlok Kumar Mishra, Wentao Bao +6
Universal multimodal embedding (UME) increasingly demands encoder's capacity for handling a broad range of tasks and modalities with increased complexity. Prior scaling methods eit…
An Event is Worth One Token: Event Tokenization for Industrial-scale LLM Recommendation
Fan Xia, Zhaoheng Zheng, Iman Setayesh +12
LLM-based recommendation has scaled along model capacity and sequence length, yet each position encodes only text, semantic IDs, or a few categorical features, discarding rich user…
TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens
Jianpeng Cheng, Xian Wu, Jiangfan Zhang +16
Recent research has demonstrated that Universal Multimodal Embedding (UME) benefits significantly from Chain-of-Thought (CoT) reasoning. In this paradigm, a generative model produc…
BLENDER: Blended Text Embeddings and Diffusion Residuals for Intra-Class Image Synthesis in Deep Metric Learning
Jan Niklas Kolf, Ozan Tezcan, Justin Theiss +7
The rise of Deep Generative Models (DGM) has enabled the generation of high-quality synthetic data. When used to augment authentic data in Deep Metric Learning (DML), these synthet…
Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views
Junyi Ma, Wentao Bao, Jingyi Xu +5
Forecasting how human hands move in egocentric views is critical for applications like augmented reality and human-robot policy transfer. Recently, several hand trajectory predicti…
Novel Diffusion Models for Multimodal 3D Hand Trajectory Prediction
Junyi Ma, Wentao Bao, Jingyi Xu +3
Predicting hand motion is critical for understanding human intentions and bridging the action space between human movements and robot manipulations. Existing hand trajectory predic…
X-SGS: Safe and Generalizable Gaussian Splatting with X-dimensional Watermarks
Zihang Cheng, Wentao Bao, Huiping Zhuang +6
3D Gaussian Splatting (3DGS) has been widely used in 3D reconstruction and 3D generation. However, the rapid adoption of 3D Gaussian Splatting raises growing concerns about informa…
MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videos
Junyi Ma, Xieyuanli Chen, Wentao Bao +2
Understanding human intentions and actions through egocentric videos is important on the path to embodied artificial intelligence. As a branch of egocentric vision techniques, hand…