Publications (57)
2.5D Visual Relationship Detection
Yu-Chuan Su, Soravit Changpinyo, Xiangning Chen +8
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Soravit Changpinyo, Piyush Sharma, Nan Ding +1
GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning
Mehran Kazemi, Hamidreza Alvari, Ankit Anand +3
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting
Su Wang, Chitwan Saharia, Ceslee Montgomery +11
Denoising Large-Scale Image Captioning from Alt-text Data using Content Selection Models
Khyathi Raghavi Chandu, Piyush Sharma, Soravit Changpinyo +2
Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling
Yaqing Wang, Jialin Wu, Tanmaya Dabral +8
Clue: Cross-modal Coherence Modeling for Caption Generation
Malihe Alikhani, Piyush Sharma, Shengjie Li +2
Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset
Ashish V. Thapliyal, Jordi Pont-Tuset, Xi Chen +1
Connecting Vision and Language with Video Localized Narratives
Paul Voigtlaender, Soravit Changpinyo, Jordi Pont-Tuset +2
Cold-Start Reinforcement Learning with Softmax Policy Gradient
Nan Ding, Radu Soricut
ImageInWords: Unlocking Hyper-Detailed Image Descriptions
Roopal Garg, Andrea Burns, Burcu Karagol Ayan +7
Improving Robust Generalization by Direct PAC-Bayesian Bound Minimization
Zifan Wang, Nan Ding, Tomer Levinboim +2
H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
Zhenhai Zhu, Radu Soricut
MaXM: Towards Multilingual Visual Question Answering
Soravit Changpinyo, Linting Xue, Michal Yarom +5
Wavelet-Based Image Tokenizer for Vision Transformers
Zhenhai Zhu, Radu Soricut
PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks
Nan Ding, Xi Chen, Tomer Levinboim +2
Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts
Jialin Wu, Xia Hu, Yaqing Wang +2
Understanding Guided Image Captioning Performance across Domains
Edwin G. Ng, Bo Pang, Piyush Sharma +1
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Zhenzhong Lan, Mingda Chen, Sebastian Goodman +3
Connecting Vision and Language with Localized Narratives
Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo +2
SHAPED: Shared-Private Encoder-Decoder for Text Style Adaptation
Ye Zhang, Nan Ding, Radu Soricut
Telling the What while Pointing to the Where: Multimodal Queries for Image Retrieval
Soravit Changpinyo, Jordi Pont-Tuset, Vittorio Ferrari +1
Gemini Robotics: Bringing AI into the Physical World
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie +115
Building Large Machine Reading-Comprehension Datasets using Paragraph Vectors
Radu Soricut, Nan Ding
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube
Jack Hessel, Zhenhai Zhu, Bo Pang +1
Informative Image Captioning with External Sources of Information
Sanqiang Zhao, Piyush Sharma, Tomer Levinboim +1
PaLI: A Jointly-Scaled Multilingual Language-Image Model
Xi Chen, Xiao Wang, Soravit Changpinyo +26
Multimodal Pretraining for Dense Video Captioning
Gabriel Huang, Bo Pang, Zhenhai Zhu +2
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Rohan Anil, Sebastian Borgeaud +1340
Attention that does not Explain Away
Nan Ding, Xinjie Fan, Zhenzhong Lan +2
A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions
Jack Hessel, Bo Pang, Zhenhai Zhu +1
Multilingual Word Embeddings using Multigraphs
Radu Soricut, Nan Ding
CausalLM is not optimal for in-context learning
Nan Ding, Tomer Levinboim, Jialin Wu +2
Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer
Gemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan +169
Reinforcing an Image Caption Generator Using Off-Line Human Feedback
Paul Hongsuck Seo, Piyush Sharma, Tomer Levinboim +2
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Anthony Brohan, Noah Brown, Justice Carbajal +51
Image Generators are Generalist Vision Learners
Valentin Gabeur, Shangbang Long, Songyou Peng +22
PaLI-3 Vision Language Models: Smaller, Faster, Stronger
Xi Chen, Xiao Wang, Lucas Beyer +16
All You May Need for VQA are Image Captions
Soravit Changpinyo, Doron Kukliansky, Idan Szpektor +3
Multi-stage Pretraining for Abstractive Summarization
Sebastian Goodman, Zhenzhong Lan, Radu Soricut
PaLI-X: On Scaling up a Multilingual Vision and Language Model
Xi Chen, Josip Djolonga, Piotr Padlewski +40
End-to-end Dense Video Captioning as Sequence Generation
Wanrong Zhu, Bo Pang, Ashish V. Thapliyal +2
Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning
Nan Ding, Xi Chen, Tomer Levinboim +2
TeaForN: Teacher-Forcing with N-grams
Sebastian Goodman, Nan Ding, Radu Soricut
Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
Nan Ding, Sebastian Goodman, Fei Sha +1
PreSTU: Pre-Training for Scene-Text Understanding
Jihyung Kil, Soravit Changpinyo, Xi Chen +4
Quality Estimation for Image Captions Based on Large-scale Human Evaluations
Tomer Levinboim, Ashish V. Thapliyal, Piyush Sharma +1
Improving Text Generation Evaluation with Batch Centering and Tempered Word Mover Distance
Xi Chen, Nan Ding, Tomer Levinboim +1
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Petko Georgiev, Ving Ian Lei +1132
PaliGemma: A versatile 3B VLM for transfer
Lucas Beyer, Andreas Steiner, André Susano Pinto +32
COSMic: A Coherence-Aware Generation Metric for Image Descriptions
Mert Ä°nan, Piyush Sharma, Baber Khalid +3
Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering
Soravit Changpinyo, Bo Pang, Piyush Sharma +1
Morpho-syntactic Lexicon Generation Using Graph-based Semi-supervised Learning
Manaal Faruqui, Ryan McDonald, Radu Soricut
Multi-Image Summarization: Textual Summary from a Set of Cohesive Images
Nicholas Trieu, Sebastian Goodman, Pradyumna Narayana +2
Cross-modal Language Generation using Pivot Stabilization for Web-scale Language Coverage
Ashish V. Thapliyal, Radu Soricut
Unified Autoregressive Visual Generation and Understanding with Continuous Tokens
Lijie Fan, Luming Tang, Siyang Qin +11
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
Gheorghe Comanici, Eric Bieber, Mike Schaekermann +3431