15 citations · 22 across the 9 of their papers we have counts for
9 papers
A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models
Noriyuki Kojima, Hadar Averbuch-Elor, Yoav Artzi
Key to tasks that require reasoning about natural language in visual contexts is grounding words and phrases to image regions. However, observing this grounding in contemporary mod…
Abstract Visual Reasoning with Tangram Shapes
Anya Ji, Noriyuki Kojima, Noah Rush +4
We introduce KiloGram, a resource for studying abstract visual reasoning in humans and machines. Drawing on the history of tangram puzzles as stimuli in cognitive science, we build…
lilGym: Natural Language Visual Reasoning with Reinforcement Learning
Anne Wu, Kianté Brantley, Noriyuki Kojima +1
We present lilGym, a new benchmark for language-conditioned reinforcement learning in visual environments. lilGym is based on 2,661 highly-compositional human-written natural langu…
Markup-to-Image Diffusion Models with Scheduled Sampling
Yuntian Deng, Noriyuki Kojima, Alexander M. Rush
Building on recent advances in image generation, we present a fully data-driven approach to rendering markup into images. The approach is based on diffusion models, which parameter…
Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior
Noriyuki Kojima, Alane Suhr, Yoav Artzi
We study continual learning for natural language instruction generation, by observing human users' instruction execution. We focus on a collaborative scenario, where the system bot…
OASIS: A Large-Scale Dataset for Single Image 3D in the Wild
Weifeng Chen, Shengyi Qian, David Fan +3
Single-view 3D is the task of recovering 3D properties such as depth and surface normals from a single image. We hypothesize that a major obstacle to single-image 3D is data. We ad…
What is Learned in Visually Grounded Neural Syntax Acquisition
Noriyuki Kojima, Hadar Averbuch-Elor, Alexander M. Rush +1
Visual features are a promising signal for learning bootstrap textual models. However, blackbox learning models make it difficult to isolate the specific contribution of visual com…
To Learn or Not to Learn: Analyzing the Role of Learning for Navigation in Virtual Environments
Noriyuki Kojima, Jia Deng
In this paper we compare learning-based methods and classical methods for navigation in virtual environments. We construct classical navigation agents and demonstrate that they out…
Speaker Naming in Movies
Mahmoud Azab, Mingzhe Wang, Max Smith +3
We propose a new model for speaker naming in movies that leverages visual, textual, and acoustic modalities in an unified optimization framework. To evaluate the performance of our…