4 papers · 1 filter
Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)
Rajveer Bachkaniwala, Chengqi Luo, Richard So +2
Context retrieval systems for LLM inference face a critical challenge: high retrieval latency creates a fundamental tension between waiting for complete context (poor time-to-first…
NEST: Network- and Memory-Aware Device Placement For Distributed Deep Learning
Irene Wang, Vishnu Varma Venkata, Arvind Krishnamurthy +1
The growing scale of deep learning demands distributed training frameworks that jointly reason about parallelism, memory, and network topology. Prior works often rely on heuristic…
VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG
Junkyum Kim, Divya Mahajan
Retrieval-Augmented Generation (RAG) systems combine vector similarity search with large language models (LLMs) to deliver accurate, context-aware responses. However, co-locating t…
MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
Seokjin Go, Divya Mahajan
Mixture-of-Experts (MoE) model architecture has emerged as a promising solution for scaling transformer models efficiently, offering sparse activation that reduces computational co…