Vectorizing string entries for data processing on tables: when are larger language models better?

Abstract

There are increasingly efficient data processing pipelines that work on vectors of numbers, for instance most machine learning models, or vector databases for fast similarity search. These require converting the data to numbers. While this conversion is easy for simple numerical and categorical entries, databases are strife with text entries, such as names or descriptions. In the age of large language models, what's the best strategies to vectorize tables entries, baring in mind that larger models entail more operational complexity? We study the benefits of language models in 14 analytical tasks on tables while varying the training size, as well as for a fuzzy join benchmark. We introduce a simple characterization of a column that reveals two settings: 1) a dirty categories setting, where strings share much similarities across entries, and conversely 2) a diverse entries setting. For dirty categories, pretrained language models bring little-to-no benefit compared to simpler string models. For diverse entries, we show that larger language models improve data processing. For these we investigate the complexity-performance tradeoffs and show that they reflect those of classic text embedding: larger models tend to perform better, but it is useful to fine tune them for embedding purposes.

References in corpus (21)

• Scikit-learn: Machine Learning in Python
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Scaling Laws for Neural Language Models
• Power Hungry Processing: Watts Driving the Cost of AI Deployment?
• Locating and Editing Factual Associations in GPT
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
• Text and Code Embeddings by Contrastive Pre-Training
• Text Embeddings by Weakly-Supervised Contrastive Pre-training
• Encoding high-cardinality string categorical variables
• C-Pack: Packed Resources For General Chinese Embeddings
• Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
• TURL: Table Understanding through Representation Learning
• TabLLM: Few-shot Classification of Tabular Data with Large Language Models
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• Language Models Represent Space and Time
• LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks
• Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
• Auto-FuzzyJoin: Auto-Program Fuzzy Similarity Joins Without Labeled Examples
• TabText: Language-Based Representations of Tabular Health Data for Predictive Modelling
• TableLlama: Towards Open Large Generalist Models for Tables