563 citations · 960 across the 10 of their papers we have counts for
4 papers · 1 filter
Rethinking embedding coupling in pre-trained language models
Hyung Won Chung, Thibault Févry, Henry Tsai +2
We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art pre-trained language models. We show that decoupled embeddings provi…
Empirical Evaluation of Pretraining Strategies for Supervised Entity Linking
Thibault Févry, Nicholas FitzGerald, Livio Baldini Soares +1
In this work, we present an entity linking model which combines a Transformer architecture with large scale pretraining from Wikipedia links. Our model achieves the state-of-the-ar…
Entities as Experts: Sparse Memory Access with Entity Supervision
Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald +2
We focus on the problem of capturing declarative knowledge about entities in the learned parameters of a language model. We introduce a new model - Entities as Experts (EAE) - that…
Learning Cross-Context Entity Representations from Text
Jeffrey Ling, Nicholas FitzGerald, Zifei Shan +4
Language modeling tasks, in which words, or word-pieces, are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent…