The effect of source disclosure on evaluation of AI-generated messages: A two-part study
arXiv:2311.15544 · doi:10.1016/j.chbah.2024.100058
Abstract
Advancements in artificial intelligence (AI) over the last decade demonstrate that machines can exhibit communicative behavior and influence how humans think, feel, and behave. In fact, the recent development of ChatGPT has shown that large language models (LLMs) can be leveraged to generate high-quality communication content at scale and across domains, suggesting that they will be increasingly used in practice. However, many questions remain about how knowing the source of the messages influences recipients' evaluation of and preference for AI-generated messages compared to human-generated messages. This paper investigated this topic in the context of vaping prevention messaging. In Study 1, which was pre-registered, we examined the influence of source disclosure on people's evaluation of AI-generated health prevention messages compared to human-generated messages. We found that source disclosure (i.e., labeling the source of a message as AI vs. human) significantly impacted the evaluation of the messages but did not significantly alter message rankings. In a follow-up study (Study 2), we examined how the influence of source disclosure may vary by the participants' negative attitudes towards AI. We found a significant moderating effect of negative attitudes towards AI on message evaluation, but not for message selection. However, for those with moderate levels of negative attitudes towards AI, source disclosure decreased the preference for AI-generated messages. Overall, the results of this series of studies showed a slight bias against AI-generated messages once the source was disclosed, adding to the emerging area of study that lies at the intersection of AI and communication.
Manuscript currently under review. Paper presented at 109th Annual National Communication Association (NCA) Conference, November 16-19, 2023. 10 pages, 5 figures. Supplementary file formatting updated in current version
References in corpus (7)
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
- Showing Academic Performance Predictions during Term Planning: Effects on Students' Decisions, Behaviors, and Preferences
- The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
- Challenges and Applications of Large Language Models
- Artificial Intelligence for Health Message Generation: Theory, Method, and an Empirical Study Using Prompt Engineering