452 citations · 469 across the 7 of their papers we have counts for
7 papers
Wavelet-Based Image Tokenizer for Vision Transformers
Zhenhai Zhu, Radu Soricut
Non-overlapping patch-wise convolution is the default image tokenizer for all state-of-the-art vision Transformer (ViT) models. Even though many ViT variants have been proposed to…
Continual Learning in Open-vocabulary Classification with Complementary Memory Systems
Zhen Zhu, Weijie Lyu, Yao Xiao +1
We introduce a method for flexible and efficient continual learning in open-vocabulary image classification, drawing inspiration from the complementary learning systems observed in…
H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
Zhenhai Zhu, Radu Soricut
We describe an efficient hierarchical method to compute attention in the Transformer architecture. The proposed attention mechanism exploits a matrix structure similar to the Hiera…
Multimodal Pretraining for Dense Video Captioning
Gabriel Huang, Bo Pang, Zhenhai Zhu +2
Learning specific hands-on skills such as cooking, car maintenance, and home repairs increasingly happens via instructional videos. The user experience with such videos is known to…
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube
Jack Hessel, Zhenhai Zhu, Bo Pang +1
Pretraining from unlabelled web videos has quickly become the de-facto means of achieving high performance on many video understanding tasks. Features are learned via prediction of…
A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions
Jack Hessel, Bo Pang, Zhenhai Zhu +1
Instructional videos get high-traffic on video sharing platforms, and prior work suggests that providing time-stamped, subtask annotations (e.g., "heat the oil in the pan") improve…
Improved Image Captioning via Policy Gradient optimization of SPIDEr
Siqi Liu, Zhenhai Zhu, Ning Ye +2
Current image captioning methods are usually trained via (penalized) maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with huma…