Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models

Abstract

The Unified Modeling Language is a standardized visual language widely used for modeling and documenting the design of software systems. Although many tools generate UML diagrams from UML code, generating executable UML code from image-based UML diagrams remains challenging. This paper proposes a new approach to generate UML code using a large multimodal language model automatically. Synthetic UML activity and sequence diagram datasets were created to train and test the model. We compared standard fine-tuning with LoRA techniques to optimize base models. The experiments measured code generation accuracy across different model sizes and training strategies. These results demonstrated that domain-adapted MM-LLMs perform for UML code generation automation, whereby, at the best model, it achieved BLEU and SSIM scores of 0.779 and 0.942 on sequence diagrams. This will enable the modernization of legacy systems and decrease the manual effort in software development workflows.

Published in the Journal of Machine Learning with Applications, Author Contributions: Averi Bates: Methodology, Development, Analysis, Data Curation, Drafting, Review. Ryan Vavricka: Data Curation, Development, Review. Shane Carleton: Supervision, Funding. Ruosi Shao: Review. Chongle Pan: Supervision, Review

References in corpus (11)

• LoRA: Low-Rank Adaptation of Large Language Models
• Evaluating Large Language Models Trained on Code
• Scaling Instruction-Finetuned Language Models
• Visual Instruction Tuning
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• CogVLM: Visual Expert for Pretrained Language Models
• LLaVA-OneVision: Easy Visual Task Transfer
• Planning with Large Language Models for Code Generation
• From Image to UML: First Results of Image Based UML Diagram Generation Using LLMs
• ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation
• Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots