123 citations · 362 across the 17 of their papers we have counts for
17 papers
Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts
Jialin Wu, Xia Hu, Yaqing Wang +2
Large multi-modal models (LMMs) exhibit remarkable performance across numerous tasks. However, generalist LMMs often suffer from performance degradation when tuned over a large col…
PaLI-3 Vision Language Models: Smaller, Faster, Stronger
Xi Chen, Xiao Wang, Lucas Beyer +16
This paper presents PaLI-3, a smaller, faster, and stronger vision language model (VLM) that compares favorably to similar models that are 10x larger. As part of arriving at this s…
Open X-Embodiment: Robotic Learning Datasets and RT-X Models
Embodiment Collaboration, Abby O'Neill, Abdul Rehman +291
Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, thi…
CausalLM is not optimal for in-context learning
Nan Ding, Tomer Levinboim, Jialin Wu +2
Recent empirical evidence indicates that transformer based in-context learning performs better when using a prefix language model (prefixLM), in which in-context samples can all at…
PaLI-X: On Scaling up a Multilingual Vision and Language Model
Xi Chen, Josip Djolonga, Piotr Padlewski +40
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training t…
Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering
Jialin Wu, Raymond J. Mooney
Most Outside-Knowledge Visual Question Answering (OK-VQA) systems employ a two-stage framework that first retrieves external knowledge given the visual question and then predicts t…
Multi-Modal Answer Validation for Knowledge-Based VQA
Jialin Wu, Jiasen Lu, Ashish Sabharwal +1
The problem of knowledge-based visual question answering involves answering questions that require external knowledge in addition to the content of the image. Such knowledge typica…
Visual Question Answering based on Local-Scene-Aware Referring Expression Generation
Jung-Jun Kim, Dong-Gyu Lee, Jialin Wu +2
Visual question answering requires a deep understanding of both images and natural language. However, most methods mainly focus on visual concept; such as the relationships between…
Improving VQA and its Explanations \\ by Comparing Competing Explanations
Jialin Wu, Liyan Chen, Raymond J. Mooney
Most recent state-of-the-art Visual Question Answering (VQA) systems are opaque black boxes that are only trained to fit the answer distribution given the question and visual conte…
Hidden State Guidance: Improving Image Captioning using An Image Conditioned Autoencoder
Jialin Wu, Raymond J. Mooney
Most RNN-based image captioning models receive supervision on the output words to mimic human captions. Therefore, the hidden states can only receive noisy gradient signals via lay…
Generating Question Relevant Captions to Aid Visual Question Answering
Jialin Wu, Zeyuan Hu, Raymond J. Mooney
Visual question answering (VQA) and image captioning require a shared body of general knowledge connecting language and vision. We present a novel approach to improve VQA performan…
Self-Critical Reasoning for Robust Visual Question Answering
Jialin Wu, Raymond J. Mooney
Visual Question Answering (VQA) deep-learning systems tend to capture superficial statistical correlations in the training data because of strong language priors and fail to genera…
Image Score: How to Select Useful Samples
Simiao Zuo, Jialin Wu
There has long been debates on how we could interpret neural networks and understand the decisions our models make. Specifically, why deep neural networks tend to be error-prone wh…
Faithful Multimodal Explanation for Visual Question Answering
Jialin Wu, Raymond J. Mooney
AI systems' ability to explain their reasoning is critical to their utility and trustworthiness. Deep neural networks have enabled significant progress on many challenging problems…
Joint Image Captioning and Question Answering
Jialin Wu, Zeyuan Hu, Raymond J. Mooney
Answering visual questions need acquire daily common knowledge and model the semantic connection among different parts in images, which is too difficult for VQA systems to learn fr…
Dynamic Filtering with Large Sampling Field for ConvNets
Jialin Wu, Dai Li, Yu Yang +2
We propose a dynamic filtering strategy with large sampling field for ConvNets (LS-DFN), where the position-specific kernels learn from not only the identical position but also mul…
Action Recognition with Joint Attention on Multi-Level Deep Features
Jialin Wu, Gu Wang, Wukui Yang +1
We propose a novel deep supervised neural network for the task of action recognition in videos, which implicitly takes advantage of visual tracking and shares the robustness of bot…