4 papers
Efficient Iterative Retrieval with Heterogeneous Batching
Dohyun Park, Hubertus Franke, Daniel G. Waddington +2
Modern information retrieval increasingly employs both embedding and generative models to handle complex queries. However, current serving systems suffer from low throughput and po…
RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems
Shaobo Li, Yirui Zhou, Yuan Xu +5
We present the design and implementation of a RAG-based AI system benchmarking (RAGPerf) framework for characterizing the system behaviors of RAG pipelines. To facilitate detailed…
Speeding up Model Loading with fastsafetensors
Takeshi Yoshimura, Tatsuhiro Chiba, Manish Sethi +2
The rapid increases in model parameter sizes introduces new challenges in pre-trained model loading. Currently, machine learning code often deserializes each parameter as a tensor…
Lossless Compression for LLM Tensor Incremental Snapshots
Daniel Waddington, Cornel Constantinescu
During the training of Large Language Models (LLMs), tensor data is periodically "checkpointed" to persistent storage to allow recovery of work done in the event of failure. The vo…