Limitations of machine learning for building energy prediction: ASHRAE Great Energy Predictor III Kaggle competition error analysis

Abstract

Research is needed to explore the limitations and potential for improvement of machine learning for building energy prediction. With this aim, the ASHRAE Great Energy Predictor III (GEPIII) Kaggle competition was launched in 2019. This effort was the largest building energy meter machine learning competition of its kind, with 4,370 participants who submitted 39,403 predictions. The test data set included two years of hourly whole building readings from 2,380 meters in 1,448 buildings at 16 locations. This paper analyzes the various sources and types of residual model error from an aggregation of the competition's top 50 solutions. This analysis reveals the limitations for machine learning using the standard model inputs of historical meter, weather, and basic building metadata. The errors are classified according to timeframe, behavior, magnitude, and incidence in single buildings or across a campus. The results show machine learning models have errors within a range of acceptability (RMSLE_scaled =< 0.1) on 79.1% of the test data. Lower magnitude (in-range) model errors (0.1 < RMSLE_scaled =< 0.3) occur in 16.1% of the test data. These errors could be remedied using innovative training data from onsite and web-based sources. Higher magnitude (out-of-range) errors (RMSLE_scaled > 0.3) occur in 4.8% of the test data and are unlikely to be accurately predicted.

References in corpus (5)

• The Building Data Genome Project 2, energy meter data from the ASHRAE Great Energy Predictor III competition
• Humans-as-a-sensor for buildings: Intensive longitudinal indoor comfort models
• The ASHRAE Great Energy Predictor III competition: Overview and results
• A Hierarchy of Limitations in Machine Learning
• Using Google Trends as a proxy for occupant behavior to predict building energy consumption

Cited by in corpus (5)

• Filling time-series gaps using image techniques: Multidimensional context autoencoder approach for building energy data imputation
• Using Google Trends as a proxy for occupant behavior to predict building energy consumption
• Interpreting Machine Learning Models for Room Temperature Prediction in Non-domestic Buildings
• Navigating Out-of-Distribution Electricity Load Forecasting during COVID-19: Benchmarking energy load forecasting models without and with continual learning
• ALDI++: Automatic and parameter-less discord and outlier detection for building energy load profiles