Balance control of a biped robot on a rotating platform based on efficient reinforcement learning

Ao Xi, Thushal Wijekoon Mudiyanselage, Dacheng Tao, Chao Chen

Research output: Contribution to journalArticleResearchpeer-review

Abstract

In this work, we combined the model based reinforcement learning (MBRL)and model free reinforcement learning (MFRL)to stabilize a biped robot (NAO robot)on a rotating platform, where the angular velocity of the platform is unknown for the proposed learning algorithm and treated as the external disturbance. Nonparametric Gaussian processes normally require a large number of training data points to deal with the discontinuity of the estimated model. Although some improved method such as probabilistic inference for learning control (PILCO)does not require an explicit global model as the actions are obtained by directly searching the policy space, the overfitting and lack of model complexity may still result in a large deviation between the prediction and the real system. Besides, none of these approaches consider the data error and measurement noise during the training process and test process, respectively. We propose a hierarchical Gaussian processes (GP)models, containing two layers of independent GPs, where the physically continuous probability transition model of the robot is obtained. Due to the physically continuous estimation, the algorithm overcomes the overfitting problem with a guaranteed model complexity, and the number of training data is also reduced. The policy for any given initial state is generated automatically by minimizing the expected cost according to the predefined cost function and the obtained probability distribution of the state. Furthermore, a novel Q (3BB)based MFRL method scheme is employed to improve the policy. Simulation results show that the proposed RL algorithm is able to balance NAO robot on a rotating platform, and it is capable of adapting to the platform with varying angular velocity.

Original languageEnglish
Article number8753751
Pages (from-to)938-951
Number of pages14
JournalIEEE/CAA Journal of Automatica Sinica
Volume6
Issue number4
DOIs
Publication statusPublished - 1 Jul 2019

Keywords

  • Biped robot
  • Gaussian processes (GP)
  • Reinforcement learning
  • Temporal difference

Cite this

Xi, Ao ; Mudiyanselage, Thushal Wijekoon ; Tao, Dacheng ; Chen, Chao. / Balance control of a biped robot on a rotating platform based on efficient reinforcement learning. In: IEEE/CAA Journal of Automatica Sinica. 2019 ; Vol. 6, No. 4. pp. 938-951.
@article{2db0e1117aaf49a381a26f593138a21b,
title = "Balance control of a biped robot on a rotating platform based on efficient reinforcement learning",
abstract = "In this work, we combined the model based reinforcement learning (MBRL)and model free reinforcement learning (MFRL)to stabilize a biped robot (NAO robot)on a rotating platform, where the angular velocity of the platform is unknown for the proposed learning algorithm and treated as the external disturbance. Nonparametric Gaussian processes normally require a large number of training data points to deal with the discontinuity of the estimated model. Although some improved method such as probabilistic inference for learning control (PILCO)does not require an explicit global model as the actions are obtained by directly searching the policy space, the overfitting and lack of model complexity may still result in a large deviation between the prediction and the real system. Besides, none of these approaches consider the data error and measurement noise during the training process and test process, respectively. We propose a hierarchical Gaussian processes (GP)models, containing two layers of independent GPs, where the physically continuous probability transition model of the robot is obtained. Due to the physically continuous estimation, the algorithm overcomes the overfitting problem with a guaranteed model complexity, and the number of training data is also reduced. The policy for any given initial state is generated automatically by minimizing the expected cost according to the predefined cost function and the obtained probability distribution of the state. Furthermore, a novel Q (3BB)based MFRL method scheme is employed to improve the policy. Simulation results show that the proposed RL algorithm is able to balance NAO robot on a rotating platform, and it is capable of adapting to the platform with varying angular velocity.",
keywords = "Biped robot, Gaussian processes (GP), Reinforcement learning, Temporal difference",
author = "Ao Xi and Mudiyanselage, {Thushal Wijekoon} and Dacheng Tao and Chao Chen",
year = "2019",
month = "7",
day = "1",
doi = "10.1109/JAS.2019.1911567",
language = "English",
volume = "6",
pages = "938--951",
journal = "IEEE/CAA Journal of Automatica Sinica",
issn = "2329-9266",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
number = "4",

}

Balance control of a biped robot on a rotating platform based on efficient reinforcement learning. / Xi, Ao; Mudiyanselage, Thushal Wijekoon; Tao, Dacheng; Chen, Chao.

In: IEEE/CAA Journal of Automatica Sinica, Vol. 6, No. 4, 8753751, 01.07.2019, p. 938-951.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Balance control of a biped robot on a rotating platform based on efficient reinforcement learning

AU - Xi, Ao

AU - Mudiyanselage, Thushal Wijekoon

AU - Tao, Dacheng

AU - Chen, Chao

PY - 2019/7/1

Y1 - 2019/7/1

N2 - In this work, we combined the model based reinforcement learning (MBRL)and model free reinforcement learning (MFRL)to stabilize a biped robot (NAO robot)on a rotating platform, where the angular velocity of the platform is unknown for the proposed learning algorithm and treated as the external disturbance. Nonparametric Gaussian processes normally require a large number of training data points to deal with the discontinuity of the estimated model. Although some improved method such as probabilistic inference for learning control (PILCO)does not require an explicit global model as the actions are obtained by directly searching the policy space, the overfitting and lack of model complexity may still result in a large deviation between the prediction and the real system. Besides, none of these approaches consider the data error and measurement noise during the training process and test process, respectively. We propose a hierarchical Gaussian processes (GP)models, containing two layers of independent GPs, where the physically continuous probability transition model of the robot is obtained. Due to the physically continuous estimation, the algorithm overcomes the overfitting problem with a guaranteed model complexity, and the number of training data is also reduced. The policy for any given initial state is generated automatically by minimizing the expected cost according to the predefined cost function and the obtained probability distribution of the state. Furthermore, a novel Q (3BB)based MFRL method scheme is employed to improve the policy. Simulation results show that the proposed RL algorithm is able to balance NAO robot on a rotating platform, and it is capable of adapting to the platform with varying angular velocity.

AB - In this work, we combined the model based reinforcement learning (MBRL)and model free reinforcement learning (MFRL)to stabilize a biped robot (NAO robot)on a rotating platform, where the angular velocity of the platform is unknown for the proposed learning algorithm and treated as the external disturbance. Nonparametric Gaussian processes normally require a large number of training data points to deal with the discontinuity of the estimated model. Although some improved method such as probabilistic inference for learning control (PILCO)does not require an explicit global model as the actions are obtained by directly searching the policy space, the overfitting and lack of model complexity may still result in a large deviation between the prediction and the real system. Besides, none of these approaches consider the data error and measurement noise during the training process and test process, respectively. We propose a hierarchical Gaussian processes (GP)models, containing two layers of independent GPs, where the physically continuous probability transition model of the robot is obtained. Due to the physically continuous estimation, the algorithm overcomes the overfitting problem with a guaranteed model complexity, and the number of training data is also reduced. The policy for any given initial state is generated automatically by minimizing the expected cost according to the predefined cost function and the obtained probability distribution of the state. Furthermore, a novel Q (3BB)based MFRL method scheme is employed to improve the policy. Simulation results show that the proposed RL algorithm is able to balance NAO robot on a rotating platform, and it is capable of adapting to the platform with varying angular velocity.

KW - Biped robot

KW - Gaussian processes (GP)

KW - Reinforcement learning

KW - Temporal difference

UR - http://www.scopus.com/inward/record.url?scp=85068790252&partnerID=8YFLogxK

U2 - 10.1109/JAS.2019.1911567

DO - 10.1109/JAS.2019.1911567

M3 - Article

VL - 6

SP - 938

EP - 951

JO - IEEE/CAA Journal of Automatica Sinica

JF - IEEE/CAA Journal of Automatica Sinica

SN - 2329-9266

IS - 4

M1 - 8753751

ER -