A Survey of Text Representation Methods and Their Genealogy
arXiv:2211.14591 · doi:10.1109/ACCESS.2022.3205719
Abstract
In recent years, with the advent of highly scalable artificial-neural-network-based text representation methods the field of natural language processing has seen unprecedented growth and sophistication. It has become possible to distill complex linguistic information of text into multidimensional dense numeric vectors with the use of the distributional hypothesis. As a consequence, text representation methods have been evolving at such a quick pace that the research community is struggling to retain knowledge of the methods and their interrelations. We contribute threefold to this lack of compilation, composition, and systematization by providing a survey of current approaches, by arranging them in a genealogy, and by conceptualizing a taxonomy of text representation methods to examine and explain the state-of-the-art. Our research is a valuable guide and reference for artificial intelligence researchers and practitioners interested in natural language processing applications such as recommender systems, chatbots, and sentiment analysis.
arXiv commentPublished online in IEEE Access
References in corpus (11)
- Efficient Estimation of Word Representations in Vector Space
- Machine learning and deep learning
- Longformer: The Long-Document Transformer
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- ERNIE: Enhanced Representation through Knowledge Integration
- Attention in Natural Language Processing
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
- CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
- Symbolic, Distributed and Distributional Representations for Natural Language Processing in the Era of Deep Learning: a Survey
- A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models
- Natural Language Processing - A Survey