### Abstract

In this work, we combined the model based reinforcement learning (MBRL)and model free reinforcement learning (MFRL)to stabilize a biped robot (NAO robot)on a rotating platform, where the angular velocity of the platform is unknown for the proposed learning algorithm and treated as the external disturbance. Nonparametric Gaussian processes normally require a large number of training data points to deal with the discontinuity of the estimated model. Although some improved method such as probabilistic inference for learning control (PILCO)does not require an explicit global model as the actions are obtained by directly searching the policy space, the overfitting and lack of model complexity may still result in a large deviation between the prediction and the real system. Besides, none of these approaches consider the data error and measurement noise during the training process and test process, respectively. We propose a hierarchical Gaussian processes (GP)models, containing two layers of independent GPs, where the physically continuous probability transition model of the robot is obtained. Due to the physically continuous estimation, the algorithm overcomes the overfitting problem with a guaranteed model complexity, and the number of training data is also reduced. The policy for any given initial state is generated automatically by minimizing the expected cost according to the predefined cost function and the obtained probability distribution of the state. Furthermore, a novel Q (3BB)based MFRL method scheme is employed to improve the policy. Simulation results show that the proposed RL algorithm is able to balance NAO robot on a rotating platform, and it is capable of adapting to the platform with varying angular velocity.

Original language | English |
---|---|

Article number | 8753751 |

Pages (from-to) | 938-951 |

Number of pages | 14 |

Journal | IEEE/CAA Journal of Automatica Sinica |

Volume | 6 |

Issue number | 4 |

DOIs | |

Publication status | Published - 1 Jul 2019 |

### Keywords

- Biped robot
- Gaussian processes (GP)
- Reinforcement learning
- Temporal difference

### Cite this

*IEEE/CAA Journal of Automatica Sinica*,

*6*(4), 938-951. [8753751]. https://doi.org/10.1109/JAS.2019.1911567

}

*IEEE/CAA Journal of Automatica Sinica*, vol. 6, no. 4, 8753751, pp. 938-951. https://doi.org/10.1109/JAS.2019.1911567

**Balance control of a biped robot on a rotating platform based on efficient reinforcement learning.** / Xi, Ao; Mudiyanselage, Thushal Wijekoon; Tao, Dacheng; Chen, Chao.

Research output: Contribution to journal › Article › Research › peer-review

TY - JOUR

T1 - Balance control of a biped robot on a rotating platform based on efficient reinforcement learning

AU - Xi, Ao

AU - Mudiyanselage, Thushal Wijekoon

AU - Tao, Dacheng

AU - Chen, Chao

PY - 2019/7/1

Y1 - 2019/7/1

N2 - In this work, we combined the model based reinforcement learning (MBRL)and model free reinforcement learning (MFRL)to stabilize a biped robot (NAO robot)on a rotating platform, where the angular velocity of the platform is unknown for the proposed learning algorithm and treated as the external disturbance. Nonparametric Gaussian processes normally require a large number of training data points to deal with the discontinuity of the estimated model. Although some improved method such as probabilistic inference for learning control (PILCO)does not require an explicit global model as the actions are obtained by directly searching the policy space, the overfitting and lack of model complexity may still result in a large deviation between the prediction and the real system. Besides, none of these approaches consider the data error and measurement noise during the training process and test process, respectively. We propose a hierarchical Gaussian processes (GP)models, containing two layers of independent GPs, where the physically continuous probability transition model of the robot is obtained. Due to the physically continuous estimation, the algorithm overcomes the overfitting problem with a guaranteed model complexity, and the number of training data is also reduced. The policy for any given initial state is generated automatically by minimizing the expected cost according to the predefined cost function and the obtained probability distribution of the state. Furthermore, a novel Q (3BB)based MFRL method scheme is employed to improve the policy. Simulation results show that the proposed RL algorithm is able to balance NAO robot on a rotating platform, and it is capable of adapting to the platform with varying angular velocity.

AB - In this work, we combined the model based reinforcement learning (MBRL)and model free reinforcement learning (MFRL)to stabilize a biped robot (NAO robot)on a rotating platform, where the angular velocity of the platform is unknown for the proposed learning algorithm and treated as the external disturbance. Nonparametric Gaussian processes normally require a large number of training data points to deal with the discontinuity of the estimated model. Although some improved method such as probabilistic inference for learning control (PILCO)does not require an explicit global model as the actions are obtained by directly searching the policy space, the overfitting and lack of model complexity may still result in a large deviation between the prediction and the real system. Besides, none of these approaches consider the data error and measurement noise during the training process and test process, respectively. We propose a hierarchical Gaussian processes (GP)models, containing two layers of independent GPs, where the physically continuous probability transition model of the robot is obtained. Due to the physically continuous estimation, the algorithm overcomes the overfitting problem with a guaranteed model complexity, and the number of training data is also reduced. The policy for any given initial state is generated automatically by minimizing the expected cost according to the predefined cost function and the obtained probability distribution of the state. Furthermore, a novel Q (3BB)based MFRL method scheme is employed to improve the policy. Simulation results show that the proposed RL algorithm is able to balance NAO robot on a rotating platform, and it is capable of adapting to the platform with varying angular velocity.

KW - Biped robot

KW - Gaussian processes (GP)

KW - Reinforcement learning

KW - Temporal difference

UR - http://www.scopus.com/inward/record.url?scp=85068790252&partnerID=8YFLogxK

U2 - 10.1109/JAS.2019.1911567

DO - 10.1109/JAS.2019.1911567

M3 - Article

VL - 6

SP - 938

EP - 951

JO - IEEE/CAA Journal of Automatica Sinica

JF - IEEE/CAA Journal of Automatica Sinica

SN - 2329-9266

IS - 4

M1 - 8753751

ER -