Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service

Abstract

This paper presents a solution to the challenge of mitigating carbon emissions from hosting large-scale machine learning (ML) inference services. ML inference is critical to modern technology products, but it is also a significant contributor to carbon footprint. We introduce Clover, a carbon-friendly ML inference service runtime system that balances performance, accuracy, and carbon emissions through mixed-quality models and GPU resource partitioning. Our experimental results demonstrate that Clover is effective in substantially reducing carbon emissions while maintaining high accuracy and meeting service level agreement (SLA) targets.

References in corpus (9)

• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Sustainable AI: Environmental Implications, Challenges and Opportunities
• Carbon Explorer: A Holistic Approach for Designing Carbon Aware Datacenters
• Clipper: A Low-Latency Online Prediction Serving System
• MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant Systems for Machine Learning
• Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
• Approximate Query Service on Autonomous IoT Cameras
• KAIROS: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resources

Cited by in corpus (4)

• A Survey on AI-driven Energy Optimisation in Terrestrial Next Generation Radio Access Networks
• The Sunk Carbon Fallacy: Rethinking Carbon Footprint Metrics for Effective Carbon-Aware Scheduling
• Loki: A System for Serving ML Inference Pipelines with Hardware and Accuracy Scaling
• Energy Efficient Scheduling of AI/ML Workloads on Multi Instance GPUs with Dynamic Repartitioning