Hessian-aware Quantized Node Embeddings for Recommendation

Abstract

Graph Neural Networks (GNNs) have achieved state-of-the-art performance in recommender systems. Nevertheless, the process of searching and ranking from a large item corpus usually requires high latency, which limits the widespread deployment of GNNs in industry-scale applications. To address this issue, many methods compress user/item representations into the binary embedding space to reduce space requirements and accelerate inference. Also, they use the Straight-through Estimator (STE) to prevent vanishing gradients during back-propagation. However, the STE often causes the gradient mismatch problem, leading to sub-optimal results. In this work, we present the Hessian-aware Quantized GNN (HQ-GNN) as an effective solution for discrete representations of users/items that enable fast retrieval. HQ-GNN is composed of two components: a GNN encoder for learning continuous node embeddings and a quantized module for compressing full-precision embeddings into low-bit ones. Consequently, HQ-GNN benefits from both lower memory requirements and faster inference speeds compared to vanilla GNNs. To address the gradient mismatch problem in STE, we further consider the quantized errors and its second-order derivatives for better stability. The experimental results on several large-scale datasets show that HQ-GNN achieves a good balance between latency and performance.

References in corpus (13)

• Neural Graph Collaborative Filtering
• Graph Convolutional Neural Networks for Web-Scale Recommender Systems
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Mixed Precision Training
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• Embedding-based Retrieval in Facebook Search
• Improving Fairness in Graph Neural Networks via Mitigating Sensitive Attribute Leakage
• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• Hashing-Accelerated Graph Neural Networks for Link Prediction
• TinyKG: Memory-Efficient Training Framework for Knowledge Graph Neural Recommender Systems