Differentiate ChatGPT-generated and Human-written Medical Texts

Abstract

Background: Large language models such as ChatGPT are capable of generating grammatically perfect and human-like text content, and a large number of ChatGPT-generated texts have appeared on the Internet. However, medical texts such as clinical notes and diagnoses require rigorous validation, and erroneous medical content generated by ChatGPT could potentially lead to disinformation that poses significant harm to healthcare and the general public. Objective: This research is among the first studies on responsible and ethical AIGC (Artificial Intelligence Generated Content) in medicine. We focus on analyzing the differences between medical texts written by human experts and generated by ChatGPT, and designing machine learning workflows to effectively detect and differentiate medical texts generated by ChatGPT. Methods: We first construct a suite of datasets containing medical texts written by human experts and generated by ChatGPT. In the next step, we analyze the linguistic features of these two types of content and uncover differences in vocabulary, part-of-speech, dependency, sentiment, perplexity, etc. Finally, we design and implement machine learning methods to detect medical text generated by ChatGPT. Results: Medical texts written by humans are more concrete, more diverse, and typically contain more useful information, while medical texts generated by ChatGPT pay more attention to fluency and logic, and usually express general terminologies rather than effective information specific to the context of the problem. A BERT-based model can effectively detect medical texts generated by ChatGPT, and the F1 exceeds 95%.

References in corpus (6)

• XGBoost: A Scalable Tree Boosting System
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
• Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics
• Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches

Cited by in corpus (3)

• Contrasting Linguistic Patterns in Human and LLM-Generated News Text
• Mapping the individual, social, and biospheric impacts of Foundation Models
• Differentiating Between Human-Written and AI-Generated Texts Using Automatically Extracted Linguistic Features