5 papers
Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving
Takeshi Yoshimura, Valentijn Dymphnus van de Beek, Tatsuhiro Chiba
Distributed LLM serving systems optimize per-request latency and throughput. However, under long-context workloads, inference accuracy becomes more variable. When incorrect respons…
Speeding up Model Loading with fastsafetensors
Takeshi Yoshimura, Tatsuhiro Chiba, Manish Sethi +2
The rapid increases in model parameter sizes introduces new challenges in pre-trained model loading. Currently, machine learning code often deserializes each parameter as a tensor…
The infrastructure powering IBM's Gen AI model development
Talia Gershon, Seetharami Seelam, Brian Belgodere +143
AI Infrastructure plays a key role in the speed and cost-competitiveness of developing and deploying advanced AI models. The current demand for powerful AI infrastructure for model…
A Robust Power Model Training Framework for Cloud Native Runtime Energy Metric Exporter
Sunyanan Choochotkaew, Chen Wang, Huamin Chen +4
Estimating power consumption in modern Cloud environments is essential for carbon quantification toward green computing. Specifically, it is important to properly account for the p…
Objcache: An Elastic Filesystem over External Persistent Storage for Container Clusters
Takeshi Yoshimura, Tatsuhiro Chiba, Sunyanan Choochotkaew +3
Container virtualization enables emerging AI workloads such as model serving, highly parallelized training, machine learning pipelines, and so on, to be easily scaled on demand on…