DAWT: Densely Annotated Wikipedia Texts across multiple languages

Abstract

In this work, we open up the DAWT dataset - Densely Annotated Wikipedia Texts across multiple languages. The annotations include labeled text mentions mapping to entities (represented by their Freebase machine ids) as well as the type of the entity. The data set contains total of 13.6M articles, 5.0B tokens, 13.8M mention entity co-occurrences. DAWT contains 4.8 times more anchor text to entity links than originally present in the Wikipedia markup. Moreover, it spans several languages including English, Spanish, Italian, German, French and Arabic. We also present the methodology used to generate the dataset which enriches Wikipedia markup in order to increase number of links. In addition to the main dataset, we open up several derived datasets including mention entity co-occurrence counts and entity embeddings, as well as mappings between Freebase ids and Wikidata item ids. We also discuss two applications of these datasets and hope that opening them up would prove useful for the Natural Language Processing and Information Retrieval communities, as well as facilitate multi-lingual research.

8 pages, 3 figures, 7 tables, WWW2017, WWW 2017 Companion proceedings

References in corpus (5)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Global Entity Ranking Across Multiple Languages
• Mining Half a Billion Topical Experts Across Multiple Social Networks
• High-Throughput and Language-Agnostic Entity Disambiguation and Linking on User Generated Data

Cited by in corpus (3)

• High-Throughput and Language-Agnostic Entity Disambiguation and Linking on User Generated Data
• Lithium NLP: A System for Rich Information Extraction from Noisy User Generated Text on Social Media
• Knowledge Graph Simple Question Answering for Unseen Domains