318 citations · 825 across the 71 of their papers we have counts for
45 papers · 1 filter
Velox: Learning Representations of 4D Geometry and Appearance
Anagh Malik, Dorian Chan, Xiaoming Zhao +3
We introduce a framework for learning latent representations of 4D objects which are descriptive, faithfully capturing object geometry and appearance; compressive, aiding in downst…
VSAS-Bench: Real-Time Evaluation of Visual Streaming Assistant Models
Pavan Kumar Anasosalu Vasu, Cem Koc, Fartash Faghri +6
Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames. This is a core mechanism for real-time vis…
LiTo: Surface Light Field Tokenization
Jen-Hao Rick Chang, Xiaoming Zhao, Dorian Chan +1
We propose a 3D latent representation that jointly models object geometry and view-dependent appearance. Most prior works focus on either reconstructing 3D geometry or predicting v…
TrajTok: Learning Trajectory Tokens enables better Video Understanding
Chenhao Zheng, Jieyu Zhang, Jianing Zhang +6
Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While…
RayRoPE: Projective Ray Positional Encoding for Multi-view Attention
Yu Wu, Minsik Jeon, Jen-Hao Rick Chang +2
We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-inv…
MobileCLIP2: Improving Multi-Modal Reinforced Training
Fartash Faghri, Pavan Kumar Anasosalu Vasu, Cem Koc +4
Foundation image-text models such as CLIP with zero-shot capabilities enable a wide array of applications. MobileCLIP is a recent family of image-text models at 3-15ms latency and…
FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations
Cheng-Yu Hsieh, Pavan Kumar Anasosalu Vasu, Fartash Faghri +5
Visual understanding is inherently contextual -- what we focus on in an image depends on the task at hand. For instance, given an image of a person holding a bouquet of flowers, we…
3D Shape Tokenization via Latent Flow Matching
Jen-Hao Rick Chang, Yuyang Wang, Miguel Angel Bautista Martin +4
We introduce a latent 3D representation that models 3D surfaces as probability density functions in 3D, i.e., p(x,y,z), with flow-matching. Our representation is specifically desig…
FastVLM: Efficient Vision Encoding for Vision Language Models
Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li +8
Scaling the input image resolution is essential for enhancing the performance of Vision Language Models (VLMs), particularly in text-rich image understanding tasks. However, popula…
Synth4Seg -- Learning Defect Data Synthesis for Defect Segmentation using Bi-level Optimization
Shancong Mou, Raviteja Vemulapalli, Shiyu Li +8
Defect segmentation is crucial for quality control in advanced manufacturing, yet data scarcity poses challenges for state-of-the-art supervised deep learning. Synthetic defect dat…
Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions
Yu-Guan Hsieh, Cheng-Yu Hsieh, Shih-Ying Yeh +7
Humans describe complex scenes with compositionality, using simple text descriptions enriched with links and relationships. While vision-language research has aimed to develop mode…
CLIP with Quality Captions: A Strong Pretraining for Vision Tasks
Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri +1
CLIP models perform remarkably well on zero-shot classification and retrieval tasks. But recent studies have shown that learnt representations in CLIP are not well suited for dense…
CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data
Sachin Mehta, Maxwell Horton, Fartash Faghri +5
Contrastive learning has emerged as a transformative method for learning effective visual representations through the alignment of image and text embeddings. However, pairwise simi…
Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications
Karren D. Yang, Anurag Ranjan, Jen-Hao Rick Chang +2
We consider the task of animating 3D facial geometry from speech signal. Existing works are primarily deterministic, focusing on learning a one-to-one mapping from speech signal to…
Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models
Raviteja Vemulapalli, Hadi Pouransari, Fartash Faghri +4
Vision Foundation Models (VFMs) pretrained on massive datasets exhibit impressive performance on various downstream tasks, especially with limited labeled target data. However, due…
MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri +2
Contrastive pretraining of image-text foundation models, such as CLIP, demonstrated excellent zero-shot performance and improved robustness on a wide range of downstream tasks. How…
TiC-CLIP: Continual Training of CLIP Models
Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari +5
Keeping large foundation models up to date on latest data is inherently expensive. To avoid the prohibitive costs of constantly retraining, it is imperative to continually train th…
SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding
Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri +6
The landscape of publicly available vision foundation models (VFMs), such as CLIP and Segment Anything Model (SAM), is expanding rapidly. VFMs are endowed with distinct capabilitie…
VISION Datasets: A Benchmark for Vision-based InduStrial InspectiON
Haoping Bai, Shancong Mou, Tatiana Likhomanenko +6
Despite progress in vision-based inspection algorithms, real-world industrial challenges -- specifically in data availability, quality, and complex production requirements -- often…
Pointersect: Neural Rendering with Cloud-Ray Intersection
Jen-Hao Rick Chang, Wei-Yu Chen, Anurag Ranjan +2
We propose a novel method that renders point clouds as if they are surfaces. The proposed method is differentiable and requires no scene-specific optimization. This unique capabili…
FaceLit: Neural 3D Relightable Faces
Anurag Ranjan, Kwang Moo Yi, Jen-Hao Rick Chang +1
We propose a generative framework, FaceLit, capable of generating a 3D face that can be rendered at various user-defined lighting conditions and views, learned purely from 2D image…
FastFill: Efficient Compatible Model Update
Florian Jaeckle, Fartash Faghri, Ali Farhadi +2
In many retrieval systems the original high dimensional data (e.g., images) is mapped to a lower dimensional feature through a learned embedding model. The task of retrieving the m…
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu +2
The recent amalgamation of transformer and convolutional designs has led to steady improvements in accuracy and efficiency of the models. In this work, we introduce FastViT, a hybr…
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement
Fartash Faghri, Hadi Pouransari, Sachin Mehta +4
We propose Dataset Reinforcement, a strategy to improve a dataset once such that the accuracy of any model architecture trained on the reinforced dataset is improved at no addition…