Can Large Language Models Grasp Concepts in Visual Content? A Case Study on YouTube Shorts about Depression

Abstract

Large language models (LLMs) are increasingly used to assist computational social science research. While prior efforts have focused on text, the potential of leveraging multimodal LLMs (MLLMs) for online video studies remains underexplored. We conduct one of the first case studies on MLLM-assisted video content analysis, comparing AI's interpretations to human understanding of abstract concepts. We leverage LLaVA-1.6 Mistral 7B to interpret four abstract concepts regarding video-mediated self-disclosure, analyzing 725 keyframes from 142 depression-related YouTube short videos. We perform a qualitative analysis of MLLM's self-generated explanations and found that the degree of operationalization can influence MLLM's interpretations. Interestingly, greater detail does not necessarily increase human-AI alignment. We also identify other factors affecting AI alignment with human understanding, such as concept complexity and versatility of video genres. Our exploratory study highlights the need to customize prompts for specific concepts and calls for researchers to incorporate more human-centered evaluations when working with AI systems in a multimodal context.

arXiv comment11 pages

References in corpus (8)

• A Survey on Multimodal Large Language Models
• Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding
• LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding
• HILL: A Hallucination Identifier for Large Language Models
• Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans
• Modeling Health Video Consumption Behaviors on Social Media: Activities, Challenges, and Characteristics
• Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression
• TempCompass: Do Video LLMs Really Understand Videos?