Product-based Neural Networks for User Response Prediction over Multi-field Categorical Data

Abstract

User response prediction is a crucial component for personalized information retrieval and filtering scenarios, such as recommender system and web search. The data in user response prediction is mostly in a multi-field categorical format and transformed into sparse representations via one-hot encoding. Due to the sparsity problems in representation and optimization, most research focuses on feature engineering and shallow modeling. Recently, deep neural networks have attracted research attention on such a problem for their high capacity and end-to-end training scheme. In this paper, we study user response prediction in the scenario of click prediction. We first analyze a coupled gradient issue in latent vector-based models and propose kernel product to learn field-aware feature interactions. Then we discuss an insensitive gradient issue in DNN-based models and propose Product-based Neural Network (PNN) which adopts a feature extractor to explore feature interactions. Generalizing the kernel product to a net-in-net architecture, we further propose Product-network In Network (PIN) which can generalize previous models. Extensive experiments on 4 industrial datasets and 1 contest dataset demonstrate that our models consistently outperform 8 baselines on both AUC and log loss. Besides, PIN makes great CTR improvement (relatively 34.67%) in online A/B test.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
• Wide & Deep Learning for Recommender Systems
• Neural Collaborative Filtering
• DeepMood: Modeling Mobile Phone Typing Dynamics for Mood Detection
• Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks
• Failures of Gradient-Based Deep Learning

Cited by in corpus (5)

• Network On Network for Tabular Data Classification in Real-world Applications
• OptEmbed: Learning Optimal Embedding Table for Click-through Rate Prediction
• Risk-Aware Bid Optimization for Online Display Advertisement
• An End-to-End Neighborhood-based Interaction Model for Knowledge-enhanced Recommendation
• Dual Graph enhanced Embedding Neural Network for CTR Prediction