The RenAIssance: Caption-Guided Diffusion Models for Historical-Art Visual Media
arXiv:2611.10599 · doi:10.1145/3841458.3841538
Abstract
Generative AI can expand cultural collections into prompt-responsive visual media, but historical artworks rarely provide the paired descriptions needed to adapt modern image generators. We present The RenAIssance, a scalable pipeline that turns a collection of historical European artworks into a reusable text-to-image media resource. The pipeline generates scene-level captions with BLIP and uses the resulting approximately 15,000 image--text pairs from DEArt to train a compact LoRA adapter for Stable Diffusion v1.5. The adapter shifts generated images toward the historical-art distribution while retaining the base model's ability to render novel concepts. On historical and contemporary prompts of different lengths, it improves FID to the DEArt reference distribution in every setting with only a modest decrease in CLIP text--image similarity. Qualitative results show that the same adapter can render both period-appropriate subjects and contemporary concepts in a coherent historical visual language. By decoupling a reusable style representation from the base generator, the approach offers a practical building block for controllable historical-art media creation, interactive authoring, and rich-media experiences.
arXiv commentAccepted at ACMMM26 RichMediaGAI. This work is co-funded by the European Union's Horizon Europe research and innovation programme under MSCA with grant agreement No 101081674
References in corpus (6)
- Auto-Encoding Variational Bayes
- IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models
- StyleAlign: Analysis and Applications of Aligned StyleGAN Models
- Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding
- PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Data
- ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style Transfer