Publications (31)
Reliable and Responsible Foundation Models: A Comprehensive Survey
Xinyu Yang, Junlin Han, Rishi Bommasani +49
Foundation models, including Large Language Models (LLMs), Multimodal Large Language Models (MLLMs), Image Generative Models (i.e, Text-to-Image Models and Image-Editing Models), a…
Strong and Controllable Blind Image Decomposition
Zeyu Zhang, Junlin Han, Chenhui Gou +2
Blind image decomposition aims to decompose all components present in an image, typically used to restore a multi-degraded input image. While fully recovering the clean image is ap…
Dual Contrastive Learning for Unsupervised Image-to-Image Translation
Junlin Han, Mehrdad Shoeiby, Lars Petersson +1
Unsupervised image-to-image translation tasks aim to find a mapping between a source domain X and a target domain Y from unpaired training data. Contrastive learning for Unpaired i…
AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting
Yuyuan Liu, Yuanhong Chen, Chong Wang +6
Segment Anything Model 2 (SAM2) exhibits strong generalisation for promptable segmentation in video clips; however, its integration with the audio modality remains underexplored. E…
Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation
Junlin Han, Jianyuan Wang, Andrea Vedaldi +2
Generating high-quality 3D content from text, single images, or sparse view images remains a challenging task with broad applications. Existing methods typically employ multi-view…
Blind Image Decomposition
Junlin Han, Weihao Li, Pengfei Fang +5
We propose and study a novel task named Blind Image Decomposition (BID), which requires separating a superimposed image into constituent underlying images in a blind setting, that…
GOSS: Towards Generalized Open-set Semantic Segmentation
Jie Hong, Weihao Li, Junlin Han +4
In this paper, we present and study a new image segmentation task, called Generalized Open-set Semantic Segmentation (GOSS). Previously, with the well-known open-set semantic segme…
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Junlin Han, Shengbang Tong, David Fan +4
Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fund…
You Only Cut Once: Boosting Data Augmentation with a Single Cut
Junlin Han, Pengfei Fang, Weihao Li +5
We present You Only Cut Once (YOCO) for performing data augmentations. YOCO cuts one image into two pieces and performs data augmentations individually within each piece. Applying…
From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images
Yiming Chen, Junlin Han, Tianyi Bai +3
While Multimodal Large Language Models (MLLMs) are adept at answering what is in an image-identifying objects and describing scenes-they often lack the ability to understand how an…
VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Models
Yufan Ren, Konstantinos Tertikas, Shalini Maiti +4
Large Vision-Language Models (LVLMs) struggle with puzzles, which require precise perception, rule comprehension, and logical reasoning. Assessing and enhancing their performance i…
Hyperbolic Audio-visual Zero-shot Learning
Jie Hong, Zeeshan Hayder, Junlin Han +3
Audio-visual zero-shot learning aims to classify samples consisting of a pair of corresponding audio and video sequences from classes that are not present during training. An analy…
DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer
Runjia Li, Junlin Han, Luke Melas-Kyriazi +6
We present DreamBeast, a novel method based on score distillation sampling (SDS) for generating fantastical 3D animal assets composed of distinct parts. Existing SDS methods often…
Learning-based Multi-View Stereo: A Survey
Fangjinhua Wang, Qingtian Zhu, Di Chang +5
3D reconstruction aims to recover the dense 3D structure of a scene. It plays an essential role in various applications such as Augmented/Virtual Reality (AR/VR), autonomous drivin…
Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Model
Zhaochong An, Guolei Sun, Yun Liu +4
Generalized few-shot 3D point cloud segmentation (GFS-PCS) adapts models to new classes with few support samples while retaining base class segmentation. Existing GFS-PCS methods e…
Scaling Sequence-to-Sequence Generative Neural Rendering
Shikun Liu, Kam Woh Ng, Wonbong Jang +9
We present Kaleido, a family of generative models designed for photorealistic, unified object- and scene-level neural rendering. Kaleido operates on the principle that 3D can be re…
EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing
Runjia Li, Moayed Haji-Ali, Ashkan Mirzaei +9
We study instruction-guided editing of egocentric videos for interactive AR applications. While recent AI video editors perform well on third-person footage, egocentric views prese…
VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models
Junlin Han, Filippos Kokkinos, Philip Torr
This paper presents a novel method for building scalable 3D generative models utilizing pre-trained video diffusion models. The primary obstacle in developing foundation 3D generat…
What Images are More Memorable to Machines?
Junlin Han, Huangying Zhan, Jie Hong +4
This paper studies the problem of measuring and predicting how memorable an image is to pattern recognition machines, as a path to explore machine intelligence. Firstly, we propose…
Curved Geometric Networks for Visual Anomaly Recognition
Jie Hong, Pengfei Fang, Weihao Li +3
Learning a latent embedding to understand the underlying nature of data distribution is often formulated in Euclidean spaces with zero curvature. However, the success of the geomet…
NeRFEditor: Differentiable Style Decomposition for Full 3D Scene Editing
Chunyi Sun, Yanbin Liu, Junlin Han +1
We present NeRFEditor, an efficient learning framework for 3D scene editing, which takes a video captured over 360° as input and outputs a high-quality, identity-preserving styliz…
How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs
Haoqin Tu, Chenhang Cui, Zijun Wang +6
This work focuses on the potential of Vision LLMs (VLLMs) in visual reasoning. Different from prior studies, we shift our focus from evaluating standard performance to introducing…
Underwater Image Restoration via Contrastive Learning and a Real-world Dataset
Junlin Han, Mehrdad Shoeiby, Tim Malthus +7
Underwater image restoration is of significant importance in unveiling the underwater world. Numerous techniques and algorithms have been developed in the past decades. However, du…
CropMix: Sampling a Rich Input Distribution via Multi-Scale Cropping
Junlin Han, Lars Petersson, Hongdong Li +1
We present a simple method, CropMix, for the purpose of producing a rich input distribution from the original dataset distribution. Unlike single random cropping, which may inadver…
Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning
Tianyi Bai, Yuxuan Fan, Jiantao Qiu +7
Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks but still struggle with fine-grained visual differences, leading to hallucination…
Small Vision-Language Models are Smart Compressors for Long Video Understanding
Junjie Fei, Jun Chen, Zechun Liu +13
Adapting Multimodal Large Language Models (MLLMs) for hour-long videos is bottlenecked by context limits. Dense visual streams saturate token budgets and exacerbate the lost-in-the…
Single Underwater Image Restoration by Contrastive Learning
Junlin Han, Mehrdad Shoeiby, Tim Malthus +6
Underwater image restoration attracts significant attention due to its importance in unveiling the underwater world. This paper elaborates on a novel method that achieves state-of-…
Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
Junlin Han, Shengbang Tong, David Fan +4
Large Language Models (LLMs), despite being trained on text alone, surprisingly develop rich visual priors. These priors allow latent visual capabilities to be unlocked for vision…
3D-GPT: Procedural 3D Modeling with Large Language Models
Chunyi Sun, Junlin Han, Weijian Deng +3
In the pursuit of efficient automated content creation, procedural generation, leveraging modifiable parameters and rule-based systems, emerges as a promising approach. Nonetheless…
Beyond Language Modeling: An Exploration of Multimodal Pretraining
Shengbang Tong, David Fan, John Nguyen +18
The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models r…
Semantic Score Distillation Sampling for Compositional Text-to-3D Generation
Ling Yang, Zixiang Zhang, Junlin Han +4
Generating high-quality 3D assets from textual descriptions remains a pivotal challenge in computer graphics and vision research. Due to the scarcity of 3D data, state-of-the-art a…