Augmenting Low-Resource Text Classification with Graph-Grounded Pre-training and Prompting

Abstract

Text classification is a fundamental problem in information retrieval with many real-world applications, such as predicting the topics of online articles and the categories of e-commerce product descriptions. However, low-resource text classification, with few or no labeled samples, poses a serious concern for supervised learning. Meanwhile, many text data are inherently grounded on a network structure, such as a hyperlink/citation network for online articles, and a user-item purchase network for e-commerce products. These graph structures capture rich semantic relationships, which can potentially augment low-resource text classification. In this paper, we propose a novel model called Graph-Grounded Pre-training and Prompting (G2P2) to address low-resource text classification in a two-pronged approach. During pre-training, we propose three graph interaction-based contrastive strategies to jointly pre-train a graph-text model; during downstream classification, we explore prompting for the jointly pre-trained model to achieve low-resource classification. Extensive experiments on four real-world datasets demonstrate the strength of G2P2 in zero- and few-shot low-resource text classification tasks.

arXiv comment11 pages, accepted by SIGIR'23

References in corpus (20)

• Efficient Estimation of Word Representations in Vector Space
• A Comprehensive Survey on Graph Neural Networks
• DeepWalk: Online Learning of Social Representations
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Learning to Prompt for Vision-Language Models
• Unsupervised Data Augmentation for Consistency Training
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
• Contrastive Learning of Medical Visual Representations from Paired Images and Text
• Open-world Learning and Application to Product Classification
• GPT-GNN: Generative Pre-Training of Graph Neural Networks
• Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
• Zero-shot Node Classification with Decomposed Graph Prototype Network
• Noisy Channel Language Model Prompting for Few-Shot Text Classification
• Meta-GNN: On Few-shot Node Classification in Graph Meta-learning
• Meta-Inductive Node Classification across Graphs
• Learning on Large-scale Text-attributed Graphs via Variational Inference
• MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators

Cited by in corpus (8)

• A Survey of Large Language Models for Graphs
• Graph Foundation Models: Concepts, Opportunities and Challenges
• A Survey on Retrieval And Structuring Augmented Generation with Large Language Models
• Pre-Training and Prompting for Few-Shot Node Classification on Text-Attributed Graphs
• A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective
• From Anchors to Answers: A Novel Node Tokenizer for Integrating Graph Structure into Large Language Models
• Text-attributed Graph Condensation via Text Selection and Attribute Matching
• Transferable and Forecastable User Targeting Foundation Model