An information-theoretic predictive model for the accuracy of AI agents adapted from psychometrics

    Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

    Abstract

    We propose a new model to quantitatively estimate the accuracy of artificial agents over cognitive tasks of approximable complexities. The model is derived by introducing notions from algorithmic information theory into a well-known (psychometric) measurement paradigm called Item Response Theory (IRT). A lower bound on accuracy can be guaranteed with respect to task complexity and the breadth of its solution space using our model. This in turn permits formulating the relationship between agent selection cost, task difficulty and accuracy as optimisation problems. Further results indicate some of the settings over which a group of cooperative agents can be more or less accurate than individual agents or other groups.

    LanguageEnglish
    Title of host publicationArtificial General Intelligence
    Subtitle of host publication10th International Conference, AGI 2017, Melbourne, VIC, Australia, August 15–18, 2017, Proceedings
    EditorsTom Everitt, Alexey Potapov, Ben Goertzel
    Place of PublicationCham, Switzerland
    PublisherSpringer-Verlag London Ltd.
    Pages225-236
    Number of pages12
    ISBN (Electronic)9783319637037
    ISBN (Print)9783319637020
    DOIs
    Publication statusPublished - 2017
    EventConference on Artificial General Intelligence (AGI) 2017 - Melbourne, Melbourne, Australia
    Duration: 15 Aug 201718 Aug 2017
    Conference number: 10th

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    PublisherSpringer
    Volume10414
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349

    Conference

    ConferenceConference on Artificial General Intelligence (AGI) 2017
    Abbreviated titleAGI 2017
    CountryAustralia
    CityMelbourne
    Period15/08/1718/08/17

    Cite this

    Chmait, N., Dowe, D. L., Li, Y-F., & Green, D. G. (2017). An information-theoretic predictive model for the accuracy of AI agents adapted from psychometrics. In T. Everitt, A. Potapov, & B. Goertzel (Eds.), Artificial General Intelligence : 10th International Conference, AGI 2017, Melbourne, VIC, Australia, August 15–18, 2017, Proceedings (pp. 225-236). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10414 ). Cham, Switzerland: Springer-Verlag London Ltd.. https://doi.org/10.1007/978-3-319-63703-7_21
    Chmait, Nader ; Dowe, David L. ; Li, Yuan-Fang ; Green, David G. / An information-theoretic predictive model for the accuracy of AI agents adapted from psychometrics. Artificial General Intelligence : 10th International Conference, AGI 2017, Melbourne, VIC, Australia, August 15–18, 2017, Proceedings. editor / Tom Everitt ; Alexey Potapov ; Ben Goertzel. Cham, Switzerland : Springer-Verlag London Ltd., 2017. pp. 225-236 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
    @inproceedings{82e4b28b707c4d87aad772fab3898d42,
    title = "An information-theoretic predictive model for the accuracy of AI agents adapted from psychometrics",
    abstract = "We propose a new model to quantitatively estimate the accuracy of artificial agents over cognitive tasks of approximable complexities. The model is derived by introducing notions from algorithmic information theory into a well-known (psychometric) measurement paradigm called Item Response Theory (IRT). A lower bound on accuracy can be guaranteed with respect to task complexity and the breadth of its solution space using our model. This in turn permits formulating the relationship between agent selection cost, task difficulty and accuracy as optimisation problems. Further results indicate some of the settings over which a group of cooperative agents can be more or less accurate than individual agents or other groups.",
    author = "Nader Chmait and Dowe, {David L.} and Yuan-Fang Li and Green, {David G.}",
    year = "2017",
    doi = "10.1007/978-3-319-63703-7_21",
    language = "English",
    isbn = "9783319637020",
    series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
    publisher = "Springer-Verlag London Ltd.",
    pages = "225--236",
    editor = "Everitt, {Tom } and Potapov, {Alexey } and Goertzel, {Ben }",
    booktitle = "Artificial General Intelligence",
    address = "Germany",

    }

    Chmait, N, Dowe, DL, Li, Y-F & Green, DG 2017, An information-theoretic predictive model for the accuracy of AI agents adapted from psychometrics. in T Everitt, A Potapov & B Goertzel (eds), Artificial General Intelligence : 10th International Conference, AGI 2017, Melbourne, VIC, Australia, August 15–18, 2017, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10414 , Springer-Verlag London Ltd., Cham, Switzerland, pp. 225-236, Conference on Artificial General Intelligence (AGI) 2017, Melbourne, Australia, 15/08/17. https://doi.org/10.1007/978-3-319-63703-7_21

    An information-theoretic predictive model for the accuracy of AI agents adapted from psychometrics. / Chmait, Nader; Dowe, David L.; Li, Yuan-Fang; Green, David G.

    Artificial General Intelligence : 10th International Conference, AGI 2017, Melbourne, VIC, Australia, August 15–18, 2017, Proceedings. ed. / Tom Everitt; Alexey Potapov; Ben Goertzel. Cham, Switzerland : Springer-Verlag London Ltd., 2017. p. 225-236 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10414 ).

    Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

    TY - GEN

    T1 - An information-theoretic predictive model for the accuracy of AI agents adapted from psychometrics

    AU - Chmait, Nader

    AU - Dowe, David L.

    AU - Li, Yuan-Fang

    AU - Green, David G.

    PY - 2017

    Y1 - 2017

    N2 - We propose a new model to quantitatively estimate the accuracy of artificial agents over cognitive tasks of approximable complexities. The model is derived by introducing notions from algorithmic information theory into a well-known (psychometric) measurement paradigm called Item Response Theory (IRT). A lower bound on accuracy can be guaranteed with respect to task complexity and the breadth of its solution space using our model. This in turn permits formulating the relationship between agent selection cost, task difficulty and accuracy as optimisation problems. Further results indicate some of the settings over which a group of cooperative agents can be more or less accurate than individual agents or other groups.

    AB - We propose a new model to quantitatively estimate the accuracy of artificial agents over cognitive tasks of approximable complexities. The model is derived by introducing notions from algorithmic information theory into a well-known (psychometric) measurement paradigm called Item Response Theory (IRT). A lower bound on accuracy can be guaranteed with respect to task complexity and the breadth of its solution space using our model. This in turn permits formulating the relationship between agent selection cost, task difficulty and accuracy as optimisation problems. Further results indicate some of the settings over which a group of cooperative agents can be more or less accurate than individual agents or other groups.

    UR - http://www.scopus.com/inward/record.url?scp=85028472248&partnerID=8YFLogxK

    U2 - 10.1007/978-3-319-63703-7_21

    DO - 10.1007/978-3-319-63703-7_21

    M3 - Conference Paper

    SN - 9783319637020

    T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

    SP - 225

    EP - 236

    BT - Artificial General Intelligence

    A2 - Everitt, Tom

    A2 - Potapov, Alexey

    A2 - Goertzel, Ben

    PB - Springer-Verlag London Ltd.

    CY - Cham, Switzerland

    ER -

    Chmait N, Dowe DL, Li Y-F, Green DG. An information-theoretic predictive model for the accuracy of AI agents adapted from psychometrics. In Everitt T, Potapov A, Goertzel B, editors, Artificial General Intelligence : 10th International Conference, AGI 2017, Melbourne, VIC, Australia, August 15–18, 2017, Proceedings. Cham, Switzerland: Springer-Verlag London Ltd. 2017. p. 225-236. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-63703-7_21