Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages
arXiv:2011.02323
Abstract
Language models based on the Transformer architecture have achieved state-of-the-art performance on a wide range of NLP tasks such as text classification, question-answering, and token classification. However, this performance is usually tested and reported on high-resource languages, like English, French, Spanish, and German. Indian languages, on the other hand, are underrepresented in such benchmarks. Despite some Indian languages being included in training multilingual Transformer models, they have not been the primary focus of such work. In order to evaluate the performance on Indian languages specifically, we analyze these language models through extensive experiments on multiple downstream tasks in Hindi, Bengali, and Telugu language. Here, we compare the efficacy of fine-tuning model parameters of pre-trained models against that of training a language model from scratch. Moreover, we empirically argue against the strict dependency between the dataset size and model performance, but rather encourage task-specific model and method selection. We achieve state-of-the-art performance on Hindi and Bengali languages for text classification task. Finally, we present effective strategies for handling the modeling of Indian languages and we release our model checkpoints for the community : https://huggingface.co/neuralspace-reverie.
Accepted at ML-RSA @ NeurIPS 2020
References in corpus (7)
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection
- AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages
- Are All Languages Created Equal in Multilingual BERT?
- BERT Based Multilingual Machine Comprehension in English and Hindi
- Experiments with POS Tagging Code-mixed Indian Social Media Text
- A POS Tagger for Code Mixed Indian Social Media Text - ICON-2016 NLP Tools Contest Entry from Surukam
Cited by in corpus (3)
- A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models
- HinFlair: pre-trained contextual string embeddings for pos tagging and text classification in the Hindi language
- Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages