141 citations · 380 across the 20 of their papers we have counts for
18 papers · 1 filter
Modality-Aware and Anatomical Vector-Quantized Autoencoding for Multimodal Brain MRI
Mingjie Li, Edward Kim, Yue Zhao +2
Learning a robust Variational Autoencoder (VAE) is a fundamental step for many deep learning applications in medical image analysis, such as MRI synthesizes. Existing brain VAEs pr…
Spherical Leech Quantization for Visual Tokenization and Generation
Yue Zhao, Hanwen Jiang, Zhenlin Xu +3
Non-parametric quantization has received much attention due to its efficiency on parameters and scalability to a large codebook. In this paper, we present a unified formulation of…
RayZer: A Self-supervised Large View Synthesis Model
Hanwen Jiang, Hao Tan, Peng Wang +8
We present RayZer, a self-supervised multi-view 3D Vision model trained without any 3D supervision, i.e., camera poses and scene geometry, while exhibiting emerging 3D awareness. C…
One-Minute Video Generation with Test-Time Training
Karan Dalal, Daniel Koceja, Gashon Hussein +12
Transformers today still struggle to generate one-minute videos because self-attention layers are inefficient for long context. Alternatives such as Mamba layers struggle with comp…
QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation
Yue Zhao, Fuzhao Xue, Scott Reed +6
We introduce Quantized Language-Image Pretraining (QLIP), a visual tokenization method that combines state-of-the-art reconstruction quality with state-of-the-art zero-shot image u…
Image and Video Tokenization with Binary Spherical Quantization
Yue Zhao, Yuanjun Xiong, Philipp Krähenbühl
We propose a new transformer-based image and video tokenizer with Binary Spherical Quantization (BSQ). BSQ projects the high-dimensional visual embedding to a lower-dimensional hyp…
VideoPrism: A Foundational Visual Encoder for Video Understanding
Long Zhao, Nitesh B. Gundavarapu, Liangzhe Yuan +16
We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus…
Distilling Vision-Language Models on Millions of Videos
Yue Zhao, Long Zhao, Xingyi Zhou +9
The recent advance in vision-language models is largely attributed to the abundance of image-text data. We aim to replicate this success for video-language models, but there simply…
LEAP: Liberate Sparse-view 3D Modeling from Camera Poses
Hanwen Jiang, Zhenyu Jiang, Yue Zhao +1
Are camera poses necessary for multi-view 3D modeling? Existing approaches predominantly assume access to accurate camera poses. While this assumption might hold for dense views, a…
Training a Large Video Model on a Single Machine in a Day
Yue Zhao, Philipp Krähenbühl
Videos are big, complex to pre-process, and slow to train on. State-of-the-art large-scale video models are trained on clusters of 32 or more GPUs for several days. As a consequenc…
Learning Video Representations from Large Language Models
Yue Zhao, Ishan Misra, Philipp Krähenbühl +1
We introduce LaViLa, a new approach to learning video-language representations by leveraging Large Language Models (LLMs). We repurpose pre-trained LLMs to be conditioned on visual…
Real-time Online Video Detection with Temporal Smoothing Transformers
Yue Zhao, Philipp Krähenbühl
Streaming video recognition reasons about objects and their actions in every frame of a video. A good streaming recognition model captures both long-term dynamics and short-term ch…
Revisiting Skeleton-based Action Recognition
Haodong Duan, Yue Zhao, Kai Chen +2
Human skeleton, as a compact representation of human action, has received increasing attention in recent years. Many skeleton-based action recognition methods adopt graph convoluti…
Intra- and Inter-Action Understanding via Temporal Action Parsing
Dian Shao, Yue Zhao, Bo Dai +1
Current methods for action recognition primarily rely on deep convolutional networks to derive feature embeddings of visual and motion features. While these methods have demonstrat…
FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding
Dian Shao, Yue Zhao, Bo Dai +1
On public benchmarks, current action recognition techniques have achieved great success. However, when used in real-world applications, e.g. sport analysis, which requires the capa…
Omni-sourced Webly-supervised Learning for Video Recognition
Haodong Duan, Yue Zhao, Yuanjun Xiong +2
We introduce OmniSource, a novel framework for leveraging web data to train video recognition models. OmniSource overcomes the barriers between data formats, such as images, short…
Temporal Action Detection with Structured Segment Networks
Yue Zhao, Yuanjun Xiong, Limin Wang +3
Detecting actions in untrimmed videos is an important yet challenging task. In this paper, we present the structured segment network (SSN), a novel framework which models the tempo…
A Pursuit of Temporal Accuracy in General Activity Detection
Yuanjun Xiong, Yue Zhao, Limin Wang +2
Detecting activities in untrimmed videos is an important but challenging task. The performance of existing methods remains unsatisfactory, e.g., they often meet difficulties in loc…