Mean Box Pooling: A Rich Image Representation and Output Embedding for the Visual Madlibs Task
arXiv:1608.02717
Abstract
We present Mean Box Pooling, a novel visual representation that pools over CNN representations of a large number, highly overlapping object proposals. We show that such representation together with nCCA, a successful multimodal embedding technique, achieves state-of-the-art performance on the Visual Madlibs task. Moreover, inspired by the nCCA's objective function, we extend classical CNN+LSTM approach to train the network by directly maximizing the similarity between the internal representation of the deep learning architecture and candidate answers. Again, such approach achieves a significant improvement over the prior work that also uses CNN+LSTM approach on Visual Madlibs.
arXiv commentAccepted to BMVC'16
References in corpus (6)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Dynamic Memory Networks for Visual and Textual Question Answering
- Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
- Stacked Attention Networks for Image Question Answering
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
- Visual Madlibs: Fill in the blank Image Generation and Question Answering