InfoTech Assistant: A Multimodal Conversational Agent for InfoTechnology Web Portal Queries

Abstract

This pilot study presents the development of the InfoTech Assistant, a domain-specific, multimodal chatbot engineered to address queries in bridge evaluation and infrastructure technology. By integrating web data scraping, large language models (LLMs), and Retrieval-Augmented Generation (RAG), the InfoTech Assistant provides accurate and contextually relevant responses. Data, including textual descriptions and images, are sourced from publicly available documents on the InfoTechnology website and organized in JSON format to facilitate efficient querying. The architecture of the system includes an HTML-based interface and a Flask back end connected to the Llama 3.1 model via LLM Studio. Evaluation results show approximately 95 percent accuracy on domain-specific tasks, with high similarity scores confirming the quality of response matching. This RAG-enhanced setup enables the InfoTech Assistant to handle complex, multimodal queries, offering both textual and visual information in its responses. The InfoTech Assistant demonstrates strong potential as a dependable tool for infrastructure professionals, delivering high accuracy and relevance in its domain-specific outputs.

Accepted by IEEE Big Data 2024

References in corpus (5)

• The Falcon Series of Open Language Models
• Is Temperature the Creativity Parameter of Large Language Models?
• Evaluating Pretrained Transformer Models for Entity Linking in Task-Oriented Dialog
• Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders
• Training and inference of large language models using 8-bit floating point