4 papers
Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning
Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer +9
Tokenization is a necessary component within the current architecture of many language mod-els, including the transformer-based large language models (LLMs) of Generative AI, yet i…
Probability-turbulence divergence: A tunable allotaxonometric instrument for comparing heavy-tailed categorical distributions
P. S. Dodds, J. R. Minot, M. V. Arnold +4
Real-world complex systems often comprise many distinct types of elements as well as many more types of networked interactions between elements. When the relative abundances of typ…
A suite of allotaxonometric tools for the comparison of complex systems using rank-turbulence divergence
Jonathan St-Onge, Ashley M. A. Fehr, Carter Ward +6
Describing and comparing complex systems requires principled, theoretically grounded tools. Built around the phenomenon of type turbulence, allotaxonographs provide map-and-list vi…
Curating corpora with classifiers: A case study of clean energy sentiment online
Michael V. Arnold, Peter Sheridan Dodds, Christopher M. Danforth
Well curated, large-scale corpora of social media posts containing broad public opinion offer an alternative data source to complement traditional surveys. While surveys are effect…