120 citations · 395 across the 27 of their papers we have counts for
28 papers
VEBench:Benchmarking Large Multimodal Models for Real-World Video Editing
Andong Deng, Dawei Du, Zhenfang Chen +7
Real-world video editing demands not only expert knowledge of cinematic techniques but also multimodal reasoning to select, align, and combine footage into coherent narratives. Whi…
SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editing
Ming Li, Xin Gu, Fan Chen +4
Due to the challenges of manually collecting accurate editing data, existing datasets are typically constructed using various automated methods, leading to noisy supervision signal…
Vidi: Large Multimodal Models for Video Understanding and Editing
Vidi Team, Celong Liu, Chia-Wen Kuo +20
Humans naturally share information with those they are connected to, and video has become one of the dominant mediums for communication and expression on the Internet. To support t…
Where do Large Vision-Language Models Look at when Answering Questions?
Xiaoying Xing, Chia-Wen Kuo, Li Fuxin +6
Large Vision-Language Models (LVLMs) have shown promising performance in vision-language understanding and reasoning tasks. However, their visual understanding behaviors remain und…
D-Attn: Decomposed Attention for Large Vision-and-Language Models
Chia-Wen Kuo, Sijie Zhu, Fan Chen +2
Large vision-and-language models (LVLMs) have traditionally integrated visual and textual tokens by concatenating them into a single homogeneous input for large language models (LL…
Multi-Reward as Condition for Instruction-based Image Editing
Xin Gu, Ming Li, Libo Zhang +4
High-quality training triplets (instruction, original image, edited image) are essential for instruction-based image editing. Predominant training datasets (e.g., InsPix2Pix) are c…
Beyond Raw Videos: Understanding Edited Videos with Large Multimodal Model
Lu Xu, Sijie Zhu, Chunyuan Li +7
The emerging video LMMs (Large Multimodal Models) have achieved significant improvements on generic video understanding in the form of VQA (Visual Question Answering), where the ra…
CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
Jiachen Li, Xinyao Wang, Sijie Zhu +6
Recent advancements in Multimodal Large Language Models (LLMs) have focused primarily on scaling by increasing text-image pair data and enhancing LLMs to improve performance on mul…
Edit3K: Universal Representation Learning for Video Editing Components
Xin Gu, Libo Zhang, Fan Chen +4
This paper focuses on understanding the predominant video creation pipeline, i.e., compositional video editing with six main types of editing components, including video effects, a…
Former: Unified etrieval and eranking Transformer for Place Recognition
Sijie Zhu, Linjie Yang, Chen Chen +3
Visual Place Recognition (VPR) estimates the location of query images by matching them with images in a reference database. Conventional methods generally adopt aggregated CNN feat…
TopNet: Transformer-based Object Placement Network for Image Compositing
Sijie Zhu, Zhe Lin, Scott Cohen +3
We investigate the problem of automatically placing an object into a background image for image compositing. Given a background image and a segmented object, the goal is to train a…
GALA: Toward Geometry-and-Lighting-Aware Object Search for Compositing
Sijie Zhu, Zhe Lin, Scott Cohen +3
Compositing-aware object search aims to find the most compatible objects for compositing given a background image and a query bounding box. Previous works focus on learning compati…
TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization
Sijie Zhu, Mubarak Shah, Chen Chen
The dominant CNN-based methods for cross-view image geo-localization rely on polar transform and fail to model global correlation. We propose a pure transformer-based approach (Tra…
BDANet: Multiscale Convolutional Neural Network with Cross-directional Attention for Building Damage Assessment from Satellite Images
Yu Shen, Sijie Zhu, Taojiannan Yang +5
Fast and effective responses are required when a natural disaster (e.g., earthquake, hurricane, etc.) strikes. Building damage assessment from satellite imagery is critical before…
MutualNet: Adaptive ConvNet via Mutual Learning from Different Model Configurations
Taojiannan Yang, Sijie Zhu, Matias Mendieta +6
Most existing deep neural networks are static, which means they can only do inference at a fixed complexity. But the resource budget can vary substantially across different devices…
3D Human Pose Estimation with Spatial and Temporal Transformers
Ce Zheng, Sijie Zhu, Matias Mendieta +3
Transformer architectures have become the model of choice in natural language processing and are now being introduced into computer vision tasks such as image classification, objec…
Consistency-based Active Learning for Object Detection
Weiping Yu, Sijie Zhu, Taojiannan Yang +1
Active learning aims to improve the performance of task model by selecting the most informative samples with a limited budget. Unlike most recent works that focused on applying act…
Deep Learning-Based Human Pose Estimation: A Survey
Ce Zheng, Wenhan Wu, Chen Chen +5
Human pose estimation aims to locate the human body parts and build human body representation (e.g., body skeleton) from input data such as images and videos. It has drawn increasi…
A3D: Adaptive 3D Networks for Video Action Recognition
Sijie Zhu, Taojiannan Yang, Matias Mendieta +1
This paper presents A3D, an adaptive 3D network that can infer at a wide range of computational constraints with one-time training. Instead of training multiple models in a grid-se…
Cross-directional Feature Fusion Network for Building Damage Assessment from Satellite Imagery
Yu Shen, Sijie Zhu, Taojiannan Yang +1
Fast and effective responses are required when a natural disaster (e.g., earthquake, hurricane, etc.) strikes. Building damage assessment from satellite imagery is critical before…
VIGOR: Cross-View Image Geo-localization beyond One-to-one Retrieval
Sijie Zhu, Taojiannan Yang, Chen Chen
Cross-view image geo-localization aims to determine the locations of street-view query images by matching with GPS-tagged reference images from aerial view. Recent works have achie…
Efficient Deep Learning of Non-local Features for Hyperspectral Image Classification
Yu Shen, Sijie Zhu, Chen Chen +4
Deep learning based methods, such as Convolution Neural Network (CNN), have demonstrated their efficiency in hyperspectral image (HSI) classification. These methods can automatical…
GradAug: A New Regularization Method for Deep Neural Networks
Taojiannan Yang, Sijie Zhu, Chen Chen
We propose a new regularization method to alleviate over-fitting in deep neural networks. The key idea is utilizing randomly transformed training samples to regularize a set of sub…
Revisiting Street-to-Aerial View Image Geo-localization and Orientation Estimation
Sijie Zhu, Taojiannan Yang, Chen Chen
Street-to-aerial image geo-localization, which matches a query street-view image to the GPS-tagged aerial images in a reference set, has attracted increasing attention recently. In…