Language Modelling with Pixels

Abstract

Language models are defined over a finite set of inputs, which creates a vocabulary bottleneck when we attempt to scale the number of supported languages. Tackling this bottleneck results in a trade-off between what can be represented in the embedding matrix and computational issues in the output layer. This paper introduces PIXEL, the Pixel-based Encoder of Language, which suffers from neither of these issues. PIXEL is a pretrained language model that renders text as images, making it possible to transfer representations across languages based on orthographic similarity or the co-activation of pixels. PIXEL is trained to reconstruct the pixels of masked patches instead of predicting a distribution over tokens. We pretrain the 86M parameter PIXEL model on the same English data as BERT and evaluate on syntactic and semantic tasks in typologically diverse languages, including various non-Latin scripts. We find that PIXEL substantially outperforms BERT on syntactic and semantic processing tasks on scripts that are not found in the pretraining data, but PIXEL is slightly weaker than BERT when working with Latin scripts. Furthermore, we find that PIXEL is more robust than BERT to orthographic attacks and linguistic code-switching, further confirming the benefits of modelling language with pixels.

ICLR 2023

References in corpus (19)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
• BEiT: BERT Pre-Training of Image Transformers
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
• CharBERT: Character-aware Pre-trained Language Model
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• SupMAE: Supervised Masked Autoencoders Are Efficient Vision Learners
• Squared English Word: A Method of Generating Glyph to Use Super Characters for Sentiment Analysis
• JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension
• Breaking Character: Are Subwords Good Enough for MRLs After All?
• Story Beyond the Eye: Glyph Positions Break PDF Text Redaction
• How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?
• Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models