Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiece

Anshul Kumar

Tokens are the basic units of Large Language Models (LLMs). LLMs rely on tokenizers to segment text into these tokens, and tokenization is the primary determinant of computational…