Script-Agnostic Language Identification

Milind Agarwal, Joshua Otten, Antonios Anastasopoulos

Language identification is used as the first step in many data collection and crawling efforts because it allows us to sort online text into language-specific buckets. However, man…