Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models

Hyunbyung Park, Sukyung Lee, Gyoungjin Gim +3

To address the challenges associated with data processing at scale, we propose Dataverse, a unified open-source Extract-Transform-Load (ETL) pipeline for large language models (LLM…