251 citations · 280 across the 9 of their papers we have counts for
9 papers
Context Unrolling in Omni Models
Ceyuan Yang, Zhijie Lin, Yang Zhao +16
We present Omni, a unified multimodal model natively trained on diverse modalities, including text, images, videos, 3D geometry, and hidden representations. We find that such train…
Unsupervised Discovery of Interpretable Directions in h-space of Pre-trained Diffusion Models
Zijian Zhang, Luping Liu, Zhijie Lin +2
We propose the first unsupervised and learning-based method to identify interpretable directions in h-space of pre-trained diffusion models. Our method is derived from an existing…
Weakly-Supervised Video Moment Retrieval via Regularized Two-Branch Proposal Networks with Erasing Mechanism
Haoyuan Li, Zhou Zhao, Zhu Zhang +1
Video moment retrieval is to identify the target moment according to the given sentence in an untrimmed video. Due to temporal boundary annotations of the video are extremely time-…
BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
Yang Zhao, Zhijie Lin, Daquan Zhou +3
LLMs have demonstrated remarkable abilities at interacting with humans through language, especially with the usage of instruction-following data. Recent advancements in LLMs, such…
DATE: Domain Adaptive Product Seeker for E-commerce
Haoyuan Li, Hao Jiang, Tao Jin +5
Product Retrieval (PR) and Grounding (PG), aiming to seek image and object-level products respectively according to a textual query, have attracted great interest recently for bett…
SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory
Zhijie Lin, Zhou Zhao, Haoyuan Li +4
Lip reading, aiming to recognize spoken sentences according to the given video of lip movements without relying on the audio stream, has attracted great interest due to its applica…
Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos
Zhu Zhang, Zhijie Lin, Zhou Zhao +1
Query-based moment retrieval aims to localize the most relevant moment in an untrimmed video according to the given natural language query. Existing works often only focus on one a…
Localizing Unseen Activities in Video via Image Query
Zhu Zhang, Zhou Zhao, Zhijie Lin +2
Action localization in untrimmed videos is an important topic in the field of video understanding. However, existing action localization methods are restricted to a pre-defined set…
Open-Ended Long-Form Video Question Answering via Hierarchical Convolutional Self-Attention Networks
Zhu Zhang, Zhou Zhao, Zhijie Lin +2
Open-ended video question answering aims to automatically generate the natural-language answer from referenced video contents according to the given question. Currently, most exist…