Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives

Abstract

This research designs a unified architecture of CTR prediction benchmark (Bench-CTR) platform that offers flexible interfaces with datasets and components of a wide range of CTR prediction models. Moreover, we construct a comprehensive system of evaluation protocols encompassing real-world and synthetic datasets, a taxonomy of metrics, standardized procedures and experimental guidelines for calibrating the performance of CTR prediction models. Furthermore, we implement the proposed benchmark platform and conduct a comparative study to evaluate a wide range of state-of-the-art models from traditional multivariate statistical to modern large language model (LLM)-based approaches on three public datasets and two synthetic datasets. Experimental results reveal that, (1) high-order models largely outperform low-order models, though such advantage varies in terms of metrics and on different datasets; (2) LLM-based models demonstrate a remarkable data efficiency, i.e., achieving the comparable performance to other models while using only 2% of the training data; (3) the performance of CTR prediction models has achieved significant improvements from 2015 to 2016, then reached a stage with slow progress, which is consistent across various datasets. This benchmark is expected to facilitate model development and evaluation and enhance practitioners' understanding of the underlying mechanisms of models in the area of CTR prediction. Code is available at https://github.com/NuriaNinja/Bench-CTR.

64 pages, 8 figures, 11 tables

References in corpus (30)

• XGBoost: A Scalable Tree Boosting System
• LoRA: Low-Rank Adaptation of Large Language Models
• xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks
• DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction
• Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction
• Causal Attention for Interpretable and Generalizable Graph Classification
• Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising
• Click-Through Rate Prediction in Online Advertising: A Literature Review
• Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks
• User Behavior Retrieval for Click-Through Rate Prediction
• Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation
• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• An Embedding Learning Framework for Numerical Features in CTR Prediction
• $F{M}^2$: Field-matrixed Factorization Machines for Recommender Systems
• CL4CTR: A Contrastive Learning Framework for CTR Prediction
• Adversarial Multimodal Representation Learning for Click-Through Rate Prediction
• EulerNet: Adaptive Feature Interaction Learning via Euler's Formula for CTR Prediction
• Click-Through Rate Prediction with Multi-Modal Hypergraphs
• Causality-based CTR Prediction using Graph Neural Networks
• Enhancing CTR Prediction with Context-Aware Feature Representation Learning
• Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space
• Deep Pattern Network for Click-Through Rate Prediction
• Hybrid CNN Based Attention with Category Prior for User Image Behavior Modeling
• AIE: Auction Information Enhanced Framework for CTR Prediction in Online Advertising
• The representation landscape of few-shot learning and fine-tuning in large language models