Attention Mechanisms in Deep Learning Models for Short-Term Energy Load Forecasting

With the spread of urbanization and climate change resulting in greater importance in energy efficiency in power grids, the use of neural networks in short-term load forecasting to make predictions on energy usage in a building has shown increased study. Common neural networks have been studied for use in Short-Term Load Forecasting (STLF) to accuracies between 80% to 95%. However, attention mechanisms in neural networks have not been thoroughly explored for use in load forecasting. Hence, this project aimed to utilize publicly available weather and energy usage datasets to compare the effects of attention in deep learning models for STLF when used to make predictions 24 hours ahead. Recurrent Neural Networks and Long Short-Term Memory models with attention implemented were compared to both simple neural networks and statistical models and several key metrics were used for comparison. With air temperature, dew temperature and wind speed used as weather data alongside date-time and historical energy usage data, it was observed that with attention implemented, both RNNs and LSTMs showed an increase in accuracy of 2.5% and 2.6%, respectively to models without attention implemented. Additionally, the results show that LSTMs with simple self-attention implemented between the input layer and the first hidden layers produced more accurate predictions than RNNs. This research and its results provide evidence that despite increases to computational cost, attention mechanisms implemented in simple deep learning models provide significant improvements to load forecast accuracy, suggesting that further improvement to these mechanisms may show additional improvements to performance.