A path to natural language through tokenisation and transformers

David S. Berman, Alexander G. Stapleton

Natural languages exhibit striking regularities in their statistical structure, including notably the emergence of Zipf's and Heaps' laws. Despite this, it remains broadly unclear…