Publications (36)
Hierarchical Self-supervised Representation Learning for Movie Understanding
Fanyi Xiao, Kaustav Kundu, Joseph Tighe +1
Most self-supervised video representation learning approaches focus on action recognition. In contrast, in this paper we focus on self-supervised video learning for movie understan…
SkeleTR: Towrads Skeleton-based Action Recognition in the Wild
Haodong Duan, Mingze Xu, Bing Shuai +4
We present SkeleTR, a new framework for skeleton-based action recognition. In contrast to prior work, which focuses mainly on controlled environments, we target more general scenar…
Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity
Zhenlin Xu, Yi Zhu, Tiffany Deng +6
This paper presents novel benchmarks for evaluating vision-language models (VLMs) in zero-shot recognition, focusing on granularity and specificity. Although VLMs excel in tasks li…
Do semantic parts emerge in Convolutional Neural Networks?
Abel Gonzalez-Garcia, Davide Modolo, Vittorio Ferrari
Semantic object parts can be useful for several visual recognition tasks. Lately, these tasks have been addressed using Convolutional Neural Networks (CNN), achieving outstanding r…
Learning Compact Video Representations for Efficient Long-form Video Understanding in Large Multimodal Models
Yuxiao Chen, Jue Wang, Zhikang Zhang +8
With recent advancements in video backbone architectures, combined with the remarkable achievements of large language models (LLMs), the analysis of long-form videos spanning tens…
TubeR: Tubelet Transformer for Video Action Detection
Jiaojiao Zhao, Yanyi Zhang, Xinyu Li +10
We propose TubeR: a simple solution for spatio-temporal video action detection. Different from existing methods that depend on either an off-line actor detector or hand-designed ac…
Combining detection and tracking for human pose estimation in videos
Manchen Wang, Joseph Tighe, Davide Modolo
We propose a novel top-down approach that tackles the problem of multi-person human pose estimation and tracking in videos. In contrast to existing top-down approaches, our method…
Multi-Object Tracking with Hallucinated and Unlabeled Videos
Daniel McKee, Bing Shuai, Andrew Berneshawi +4
In this paper, we explore learning end-to-end deep neural trackers without tracking annotations. This is important as large-scale training data is essential for training deep neura…
Early Action Recognition with Action Prototypes
Guglielmo Camporese, Alessandro Bergamo, Xunyu Lin +2
Early action recognition is an important and challenging problem that enables the recognition of an action from a partially observed video stream where the activity is potentially…
Denoising and Selecting Pseudo-Heatmaps for Semi-Supervised Human Pose Estimation
Zhuoran Yu, Manchen Wang, Yanbei Chen +2
We propose a new semi-supervised learning design for human pose estimation that revisits the popular dual-student framework and enhances it two ways. First, we introduce a denoisin…
Multi-Object Tracking with Siamese Track-RCNN
Bing Shuai, Andrew G. Berneshawi, Davide Modolo +1
Multi-object tracking systems often consist of a combination of a detector, a short term linker, a re-identification feature extractor and a solver that takes the output from these…
SCVRL: Shuffled Contrastive Video Representation Learning
Michael Dorkenwald, Fanyi Xiao, Biagio Brattoli +2
We propose SCVRL, a novel contrastive-based framework for self-supervised learning for videos. Differently from previous contrast learning based methods that mostly focus on learni…
Learning Semantic Part-Based Models from Google Images
Davide Modolo, Vittorio Ferrari
We propose a technique to train semantic part-based models of object classes from Google Images. Our models encompass the appearance of parts and their spatial arrangement on the o…
ScaleDet: A Scalable Multi-Dataset Object Detector
Yanbei Chen, Manchen Wang, Abhay Mittal +4
Multi-dataset training provides a viable solution for exploiting heterogeneous large-scale datasets without extra annotation cost. In this work, we propose a scalable multi-dataset…
Action recognition with spatial-temporal discriminative filter banks
Brais Martinez, Davide Modolo, Yuanjun Xiong +1
Action recognition has seen a dramatic performance improvement in the last few years. Most of the current state-of-the-art literature either aims at improving performance through c…
What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactions
A S M Iftekhar, Hao Chen, Kaustav Kundu +3
We propose a novel one-stage Transformer-based semantic and spatial refined transformer (SSRT) to solve the Human-Object Interaction detection task, which requires to localize huma…
Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs
Zhenhong Sun, Hanqing Zhao, Yatao Bian +7
Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive LLMs, offering efficient generation through block-wise progressive unmasking. Howe…
SiamMOT: Siamese Multi-Object Tracking
Bing Shuai, Andrew Berneshawi, Xinyu Li +2
In this paper, we focus on improving online multi-object tracking (MOT). In particular, we introduce a region-based Siamese Multi-Object Tracking network, which we name SiamMOT. Si…
Visual Reasoning through Tool-supervised Reinforcement Learning
Qihua Dong, Gozde Sahin, Pei Wang +4
In this paper, we investigate the problem of how to effectively master tool-use to solve complex visual reasoning tasks for Multimodal Large Language Models. To achieve that, we pr…
Hyperbolic Learning with Synthetic Captions for Open-World Detection
Fanjie Kong, Yanbei Chen, Jiarui Cai +1
Open-world detection poses significant challenges, as it requires the detection of any object using either object class labels or free-form texts. Existing related works often use…
MaCLR: Motion-aware Contrastive Learning of Representations for Videos
Fanyi Xiao, Joseph Tighe, Davide Modolo
We present MaCLR, a novel method to explicitly perform cross-modal self-supervised video representations learning from visual and motion modalities. Compared to previous video repr…
Context Forest for efficient object detection with large mixture models
Davide Modolo, Alexander Vezhnevets, Vittorio Ferrari
We present Context Forest (ConF), a technique for predicting properties of the objects in an image based on its global appearance. Compared to standard nearest-neighbour techniques…
The Amazon Nova Family of Models: Technical Report and Model Card
Amazon AGI, Aaron Langford, Aayush Shah +783
We present Amazon Nova, a new generation of state-of-the-art foundation models that deliver frontier intelligence and industry-leading price performance. Amazon Nova Pro is a highl…
Understanding the impact of mistakes on background regions in crowd counting
Davide Modolo, Bing Shuai, Rahul Rama Varior +1
Every crowd counting researcher has likely observed their model output wrong positive predictions on image regions not containing any person. But how often do these mistakes happen…
Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts
Zhaoyang Zhang, Yantao Shen, Kunyu Shi +7
We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks which may interfere with each other, result…
STORM: End-to-End Referring Multi-Object Tracking in Videos
Zijia Lu, Jingru Yi, Jue Wang +4
Referring multi-object tracking (RMOT) is a task of associating all the objects in a video that semantically match with given textual queries or referring expressions. Existing RMO…
Semi-supervised Vision Transformers at Scale
Zhaowei Cai, Avinash Ravichandran, Paolo Favaro +5
We study semi-supervised learning (SSL) for vision transformers (ViT), an under-explored topic despite the wide adoption of the ViT architectures to different tasks. To tackle this…
Selective Feature Compression for Efficient Activity Recognition Inference
Chunhui Liu, Xinyu Li, Hao Chen +2
Most action recognition solutions rely on dense sampling to precisely cover the informative temporal clip. Extensively searching temporal region is expensive for a real-world appli…
Objects as context for detecting their semantic parts
Abel Gonzalez-Garcia, Davide Modolo, Vittorio Ferrari
We present a semantic part detection approach that effectively leverages object information.We use the object appearance and its class as indicators of what parts to expect. We als…
Multi-Scale Attention Network for Crowd Counting
Rahul Rama Varior, Bing Shuai, Joseph Tighe +1
In crowd counting datasets, people appear at different scales, depending on their distance from the camera. To address this issue, we propose a novel multi-branch scale-aware atten…
Mural: Transferring LLM knowledge to image generation via Mixture-of-Transformers
Achin Jain, Jie An, Siddharth Chaudhary +1
Leveraging capabilities of large language models (LLMs) in text-to-image (T2I) synthesis is an important research direction. In this work we investigate whether the knowledge of a…
MM-ReCoder: Advancing Chart-to-Code Generation with Reinforcement Learning and Self-Correction
Zitian Tang, Xu Zhang, Jianbo Yuan +4
Multimodal Large Language Models (MLLMs) have recently demonstrated promising capabilities in multimodal coding tasks such as chart-to-code generation. However, existing methods pr…
Self-Supervised Multi-Object Tracking with Path Consistency
Zijia Lu, Bing Shuai, Yanbei Chen +2
In this paper, we propose a novel concept of path consistency to learn robust object matching without using manual object identity supervision. Our key idea is that, to track a obj…
Transfer of Representations to Video Label Propagation: Implementation Factors Matter
Daniel McKee, Zitong Zhan, Bing Shuai +3
This work studies feature representations for dense label propagation in video, with a focus on recently proposed methods that learn video correspondence using self-supervised sign…
Joint calibration of Ensemble of Exemplar SVMs
Davide Modolo, Alexander Vezhnevets, Olga Russakovsky +1
We present a method for calibrating the Ensemble of Exemplar SVMs model. Unlike the standard approach, which calibrates each SVM independently, our method optimizes their joint per…
SemiGPC: Distribution-Aware Label Refinement for Imbalanced Semi-Supervised Learning Using Gaussian Processes
Abdelhak Lemkhenter, Manchen Wang, Luca Zancato +3
In this paper we introduce SemiGPC, a distribution-aware label refinement strategy based on Gaussian Processes where the predictions of the model are derived from the labels poster…