Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

Abstract

We introduce a model for constructing vector representations of words by composing characters using bidirectional LSTMs. Relative to traditional word representation models that have independent vectors for each word type, our model requires only a single vector per character type and a fixed set of parameters for the compositional model. Despite the compactness of this model and, more importantly, the arbitrary nature of the form-function relationship in language, our "composed" word representations yield state-of-the-art results in language modeling and part-of-speech tagging. Benefits over traditional baselines are particularly pronounced in morphologically rich languages (e.g., Turkish).

References in corpus (2)

• Natural Language Processing (almost) from Scratch
• Structure Regularization for Structured Prediction: Theories and Experiments

Cited by in corpus (50)

• Character-Aware Neural Language Models
• Exploring the Limits of Language Modeling
• Character-based Neural Machine Translation
• Natural Language Processing Advancements By Deep Learning: A Survey
• Multi-Task Cross-Lingual Sequence Tagging from Scratch
• Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers
• Latent Predictor Networks for Code Generation
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Learning to Compute Word Embeddings On the Fly
• A Survey on Transfer Learning in Natural Language Processing
• Gated Word-Character Recurrent Language Model
• Words or Characters? Fine-grained Gating for Reading Comprehension
• RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model
• Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks
• Recurrent Neural Network Language Models for Open Vocabulary Event-Level Cyber Anomaly Detection
• Multilingual Language Processing From Bytes
• Contextual Word Representations: A Contextual Introduction
• Investigating Bi-LSTM and CRF with POS Tag Embedding for Indonesian Named Entity Tagger
• Neural Morphological Tagging from Characters for Morphologically Rich Languages
• Learning to Represent Words in Context with Multilingual Supervision
• A Survey on Recent Advances in Sequence Labeling from Deep Learning Models
• ner and pos when nothing is capitalized
• Enriching Rare Word Representations in Neural Language Models by Embedding Matrix Augmentation
• Robust Multilingual Part-of-Speech Tagging via Adversarial Training
• An Empirical Exploration of Skip Connections for Sequential Tagging
• Character-Level Language Modeling with Hierarchical Recurrent Neural Networks
• Evaluating Models of Robust Word Recognition with Serial Reproduction
• Deep Semi-Supervised Learning with Linguistically Motivated Sequence Labeling Task Hierarchies
• Data Augmentation via Dependency Tree Morphing for Low-Resource Languages
• An Efficient Character-Level Neural Machine Translation
• A Convolutional Neural Network for Language-Agnostic Source Code Summarization
• A Character-Word Compositional Neural Language Model for Finnish
• Read, Tag, and Parse All at Once, or Fully-neural Dependency Parsing
• Shortcut Sequence Tagging
• Neural Architectures for Named Entity Recognition
• Translating Natural Language to SQL using Pointer-Generator Networks and How Decoding Order Matters
• Character-based Neural Networks for Sentence Pair Modeling
• Authorship Attribution in Bangla literature using Character-level CNN
• Attending Form and Context to Generate Specialized Out-of-VocabularyWords Representations
• Character-level Deep Conflation for Business Data Analytics
• Strawman: an Ensemble of Deep Bag-of-Ngrams for Sentiment Analysis
• Nonsymbolic Text Representation
• Bi-directional LSTM Recurrent Neural Network for Chinese Word Segmentation
• UnibucKernel: Geolocating Swiss German Jodels Using Ensemble Learning
• End-to-end Graph-based TAG Parsing with Neural Networks
• Czech Text Processing with Contextual Embeddings: POS Tagging, Lemmatization, Parsing and NER
• Compositional Sentence Representation from Character within Large Context Text
• Character-Level Models versus Morphology in Semantic Role Labeling
• Comparison of Turkish Word Representations Trained on Different Morphological Forms
• Syllable-level Neural Language Model for Agglutinative Language