Contextual Word Representations: A Contextual Introduction

Abstract

This introduction aims to tell the story of how we put words into computers. It is part of the story of the field of natural language processing (NLP), a branch of artificial intelligence. It targets a wide audience with a basic understanding of computer programming, but avoids a detailed mathematical treatment, and it does not present any algorithms. It also does not focus on any particular application of NLP such as translation, question answering, or information extraction. The ideas presented here were developed by many researchers over many decades, so the citations are not exhaustive but rather direct the reader to a handful of papers that are, in the author's view, seminal. After reading this document, you should have a general understanding of word vectors (also known as word embeddings): why they exist, what problems they solve, where they come from, how they have changed over time, and what some of the open questions about them are. Readers already familiar with word vectors are advised to skip to Section 5 for the discussion of the most recent advance, contextual word vectors.

References in corpus (6)

• Natural Language Processing (almost) from Scratch
• Semantics derived automatically from language corpora contain human-like biases
• From Frequency to Meaning: Vector Space Models of Semantics
• Cross-lingual Language Model Pretraining
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• Assessing BERT's Syntactic Abilities

Cited by in corpus (7)

• Cultural Cartography with Word Embeddings
• Protection of the patient data against intentional attacks using a hybrid robust watermarking code
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
• Which *BERT? A Survey Organizing Contextualized Encoders
• Exploring the Combination of Contextual Word Embeddings and Knowledge Graph Embeddings
• Tha3aroon at NSURL-2019 Task 8: Semantic Question Similarity in Arabic
• Language Models as Zero-shot Visual Semantic Learners