Multilingual Hierarchical Attention Networks for Document Classification

Abstract

Hierarchical attention networks have recently achieved remarkable performance for document classification in a given language. However, when multilingual document collections are considered, training such models separately for each language entails linear parameter growth and lack of cross-language transfer. Learning a single multilingual model with fewer parameters is therefore a challenging but potentially beneficial objective. To this end, we propose multilingual hierarchical attention networks for learning document structures, with shared encoders and/or shared attention mechanisms across languages, using multi-task learning and an aligned semantic space as input. We evaluate the proposed models on multilingual document classification with disjoint label sets, on a large dataset which we provide, with 600k news documents in 8 languages, and 5k labels. The multilingual models outperform monolingual ones in low-resource as well as full-resource settings, and use fewer parameters, thus confirming their computational efficiency and the utility of cross-language transfer.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Effective Approaches to Attention-based Neural Machine Translation
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
• Polyglot: Distributed Word Representations for Multilingual NLP
• Massively Multilingual Word Embeddings
• Multilingual Models for Compositional Distributed Semantics

Cited by in corpus (2)

• A3T-GCN: Attention Temporal Graph Convolutional Network for Traffic Forecasting
• Short Text Classification via Knowledge powered Attention with Similarity Matrix based CNN