AutoCaption: Image Captioning with Neural Architecture Search
arXiv:2012.09742
Abstract
Image captioning transforms complex visual information into abstract natural language for representation, which can help computers understanding the world quickly. However, due to the complexity of the real environment, it needs to identify key objects and realize their connections, and further generate natural language. The whole process involves a visual understanding module and a language generation module, which brings more challenges to the design of deep neural networks than other tasks. Neural Architecture Search (NAS) has shown its important role in a variety of image recognition tasks. Besides, RNN plays an essential role in the image captioning task. We introduce a AutoCaption method to better design the decoder module of the image captioning where we use the NAS to design the decoder module called AutoRNN automatically. We use the reinforcement learning method based on shared parameters for automatic design the AutoRNN efficiently. The search space of the AutoCaption includes connections between the layers and the operations in layers both, and it can make AutoRNN express more architectures. In particular, RNN is equivalent to a subset of our search space. Experiments on the MSCOCO datasets show that our AutoCaption model can achieve better performance than traditional hand-design methods.
References in corpus (14)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- LSTM: A Search Space Odyssey
- Neural Architecture Search with Reinforcement Learning
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Efficient Neural Architecture Search via Parameter Sharing
- Large-Scale Evolution of Image Classifiers
- Designing Neural Network Architectures using Reinforcement Learning
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- The Evolved Transformer