lamBERT: Language and Action Learning Using Multimodal BERT
arXiv:2004.07093
Abstract
Recently, the bidirectional encoder representations from transformers (BERT) model has attracted much attention in the field of natural language processing, owing to its high performance in language understanding-related tasks. The BERT model learns language representation that can be adapted to various tasks via pre-training using a large corpus in an unsupervised manner. This study proposes the language and action learning using multimodal BERT (lamBERT) model that enables the learning of language and actions by 1) extending the BERT model to multimodal representation and 2) integrating it with reinforcement learning. To verify the proposed model, an experiment is conducted in a grid environment that requires language understanding for the agent to act properly. As a result, the lamBERT model obtained higher rewards in multitask settings and transfer settings when compared to other models, such as the convolutional neural network-based model and the lamBERT model without pre-training.
8 pages, 9 figures
References in corpus (5)
- Reformer: The Efficient Transformer
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
- Grounded Language Learning in a Simulated 3D World
- Joint Multimodal Learning with Deep Generative Models
- Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models