Socially Aware Synthetic Data Generation for Suicidal Ideation Detection Using Large Language Models

Abstract

Suicidal ideation detection is a vital research area that holds great potential for improving mental health support systems. However, the sensitivity surrounding suicide-related data poses challenges in accessing large-scale, annotated datasets necessary for training effective machine learning models. To address this limitation, we introduce an innovative strategy that leverages the capabilities of generative AI models, such as ChatGPT, Flan-T5, and Llama, to create synthetic data for suicidal ideation detection. Our data generation approach is grounded in social factors extracted from psychology literature and aims to ensure coverage of essential information related to suicidal ideation. In our study, we benchmarked against state-of-the-art NLP classification models, specifically, those centered around the BERT family structures. When trained on the real-world dataset, UMD, these conventional models tend to yield F1-scores ranging from 0.75 to 0.87. Our synthetic data-driven method, informed by social factors, offers consistent F1-scores of 0.82 for both models, suggesting that the richness of topics in synthetic data can bridge the performance gap across different model complexities. Most impressively, when we combined a mere 30% of the UMD dataset with our synthetic data, we witnessed a substantial increase in performance, achieving an F1-score of 0.88 on the UMD test set. Such results underscore the cost-effectiveness and potential of our approach in confronting major challenges in the field, such as data scarcity and the quest for diversity in data representation.

References in corpus (8)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Scaling Instruction-Finetuned Language Models
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• Towards Interpretable Mental Health Analysis with Large Language Models
• From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer
• PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model
• Suicidal Ideation Detection on Social Media: A Review of Machine Learning Methods

Cited by in corpus (5)

• A Scoping Review of Synthetic Data Generation by Language Models in Biomedical Research and Application: Data Utility and Quality Perspectives
• Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agents
• Delving Into the Psychology of Machines: Exploring the Structure of Self-Regulated Learning via LLM-Generated Survey Responses
• Evaluating Transformer Models for Suicide Risk Detection on Social Media
• Transferring Natural Language Datasets Between Languages Using Large Language Models for Modern Decision Support and Sci-Tech Analytical Systems