Batch Prompting: Efficient Inference with Large Language Model APIs

Zhoujun Cheng, Jungo Kasai, Tao Yu

Performing inference on large volumes of samples with large language models (LLMs) can be computationally and financially costly in industry and real-world use. We propose batch pr…