Speed and Conversational Large Language Models: Not All Is About Tokens per Second

Abstract

The speed of open-weights large language models (LLMs) and its dependency on the task at hand, when run on GPUs, is studied to present a comparative analysis of the speed of the most popular open LLMs.

References in corpus (5)

• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Gemini: A Family of Highly Capable Multimodal Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Cited by in corpus (1)

• Thoughtful, Confused, or Untrustworthy: How Text Presentation Influences Perceptions of AI Writing Tools