Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages

Abstract

Language models based on the Transformer architecture have achieved state-of-the-art performance on a wide range of NLP tasks such as text classification, question-answering, and token classification. However, this performance is usually tested and reported on high-resource languages, like English, French, Spanish, and German. Indian languages, on the other hand, are underrepresented in such benchmarks. Despite some Indian languages being included in training multilingual Transformer models, they have not been the primary focus of such work. In order to evaluate the performance on Indian languages specifically, we analyze these language models through extensive experiments on multiple downstream tasks in Hindi, Bengali, and Telugu language. Here, we compare the efficacy of fine-tuning model parameters of pre-trained models against that of training a language model from scratch. Moreover, we empirically argue against the strict dependency between the dataset size and model performance, but rather encourage task-specific model and method selection. We achieve state-of-the-art performance on Hindi and Bengali languages for text classification task. Finally, we present effective strategies for handling the modeling of Indian languages and we release our model checkpoints for the community : https://huggingface.co/neuralspace-reverie.

arXiv commentAccepted at ML-RSA @ NeurIPS 2020

References in corpus (14)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• CamemBERT: a Tasty French Language Model
• Enriching Word Vectors with Subword Information
• Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection
• Sentiment Analysis of Code-Mixed Indian Languages: An Overview of SAIL_Code-Mixed Shared Task @ICON-2017
• AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages
• Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing
• TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
• Are All Languages Created Equal in Multilingual BERT?
• BERT Based Multilingual Machine Comprehension in English and Hindi
• Experiments with POS Tagging Code-mixed Indian Social Media Text
• A POS Tagger for Code Mixed Indian Social Media Text - ICON-2016 NLP Tools Contest Entry from Surukam

Cited by in corpus (3)

• A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models
• HinFlair: pre-trained contextual string embeddings for pos tagging and text classification in the Hindi language
• Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages