Proximal policy optimization based reinforcement learning for joint bidding in energy and frequency regulation markets

Research output: Chapter in Book/Report/Conference proceedingConference PaperOther

6 Citations (Scopus)


Driven by the global decarbonization effort, the rapid integration of renewable energy into the conventional electricity grid presents new challenges and opportunities for the battery energy storage system (BESS) participating in the energy market. Energy arbitrage can be a significant source of revenue for the BESS due to the increasing price volatility in the spot market caused by the mismatch between renewable generation and electricity demand. In addition, the Frequency Control Ancillary Services (FCAS) markets established to stabilize the grid can offer higher returns for the BESS due to their capability to respond within milliseconds. Therefore, it is crucial for the BESS to carefully decide how much capacity to assign to each market to maximize the total profit under uncertain market conditions. This paper formulates the bidding problem of the BESS as a Markov Decision Process, which enables the BESS to participate in both the spot market and the FCAS market to maximize profit. Then, Proximal Policy Optimization, a model-free deep reinforcement learning algorithm, is employed to learn the optimal bidding strategy from the dynamic environment of the energy market under a continuous bidding scale. The proposed model is trained and validated using real-world historical data of the Australian National Electricity Market. The results demonstrate that our developed joint bidding strategy in both markets is significantly profitable compared to individual markets.

Original languageEnglish
Title of host publication2022 IEEE Power and Energy Society General Meeting, PESGM 2022
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages5
ISBN (Electronic)9781665408233
ISBN (Print)9781665408240
Publication statusPublished - 2022
EventIEEE Power and Energy Society General Meeting 2022 - Denver, United States of America
Duration: 17 Jul 202221 Jul 2022 (Proceedings)

Publication series

NameIEEE Power and Energy Society General Meeting
PublisherIEEE, Institute of Electrical and Electronics Engineers
ISSN (Print)1944-9925
ISSN (Electronic)1944-9933


ConferenceIEEE Power and Energy Society General Meeting 2022
Abbreviated titlePESGM 2022
Country/TerritoryUnited States of America
Internet address

Cite this