Exploring Adapter-based Transfer Learning for Recommender Systems: Empirical Studies and Practical Insights

Abstract

Adapters, a plug-in neural network module with some tunable parameters, have emerged as a parameter-efficient transfer learning technique for adapting pre-trained models to downstream tasks, especially for natural language processing (NLP) and computer vision (CV) fields. Meanwhile, learning recommendation models directly from raw item modality features -- e.g., texts of NLP and images of CV -- can enable effective and transferable recommender systems (called TransRec). In view of this, a natural question arises: can adapter-based learning techniques achieve parameter-efficient TransRec with good performance? To this end, we perform empirical studies to address several key sub-questions. First, we ask whether the adapter-based TransRec performs comparably to TransRec based on standard full-parameter fine-tuning? does it hold for recommendation with different item modalities, e.g., textual RS and visual RS. If yes, we benchmark these existing adapters, which have been shown to be effective in NLP and CV tasks, in item recommendation tasks. Third, we carefully study several key factors for the adapter-based TransRec in terms of where and how to insert these adapters? Finally, we look at the effects of adapter-based TransRec by either scaling up its source training data or scaling down its target training data. Our paper provides key insights and practical guidance on unified & transferable recommendation -- a less studied recommendation scenario. We release our codes and other materials at: https://github.com/westlake-repl/Adapter4Rec/.

arXiv commentAccepted by WSDM2024

References in corpus (35)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Representation Learning with Contrastive Predictive Coding
• LoRA: Low-Rank Adaptation of Large Language Models
• On the Opportunities and Risks of Foundation Models
• Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering
• TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation
• BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• Multi-Modal Self-Supervised Learning for Recommendation
• Vision Transformer Adapter for Dense Predictions
• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
• Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning
• LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• Prompt Learning for News Recommendation
• MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation
• Recommender Systems with Generative Retrieval
• M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems
• One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction
• One Person, One Model, One World: Learning Continual User Representation without Forgetting
• Zero-Shot Recommender Systems
• Parameter-efficient Model Adaptation for Vision Transformers
• MM-Rec: Multimodal News Recommendation
• TransRec: Learning Transferable Recommendation from Mixture-of-Modality Feedback
• Scattered or Connected? An Optimized Parameter-efficient Tuning Approach for Information Retrieval
• End-to-end Learnable Diversity-aware News Recommendation
• Rethinking Reinforcement Learning for Recommendation: A Prompt Perspective
• Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)
• Prompt Tuning Large Language Models on Personalized Aspect Extraction for Recommendations
• Two Birds with One Stone: Unified Model Learning for Both Recall and Ranking in News Recommendation
• Thoroughly Modeling Multi-domain Pre-trained Recommendation as Language
• A Content-Driven Micro-Video Recommendation Dataset at Scale
• ID Embedding as Subtle Features of Content and Structure for Multimodal Recommendation
• NineRec: A Benchmark Dataset Suite for Evaluating Transferable Recommendation
• Collaborative Word-based Pre-trained Item Representation for Transferable Recommendation
• Personalized Prompt for Sequential Recommendation

Cited by in corpus (8)

• Recommender Systems in the Era of Large Language Models (LLMs)
• When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities
• IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT
• Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning
• ABXI: Invariant Interest Adaptation for Task-Guided Cross-Domain Sequential Recommendation
• Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation
• X-Cross: Dynamic Integration of Language Models for Cross-Domain Sequential Recommendation
• Exploring the Upper Limits of Text-Based Collaborative Filtering Using Large Language Models: Discoveries and Insights