A dynamic intelligence test framework for evaluating AI agents

Nader Chmait, Yuan-Fang Li, David L. Dowe, David Green

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearch


In our recent work on the measurement of (collective) intelligence,
we used a dynamic intelligence test to measure and compare
the performances of artificial agents. In this paper we give a detailed technical description of the testing framework, its design and implementation, showing how it can be used to quantitatively evaluate general purpose, single- and multi-agent artificial intelligence (AI). The source code and scripts to run experiments have been released as open-source, and instructions on how to administer the test to artificial agents have been outlined. This will allow evaluating new agent behaviours and also extending the scope of the test. Alternative
testing environments are discussed along with other considerations
relevant to the robustness of multi-agent performance tests.
The intuition is to encourage people in the AI community to quantitatively
evaluate new types of heuristics and algorithms individually
and collectively using different communication and interaction protocols,
and thus pave the way towards a rigorous, formal and unified
testing framework for general purpose agents.
Original languageEnglish
Title of host publicationEGPAI 2016 - Evaluating General Purpose AI 2016
EditorsChristos Dimitrakakis, Jose Hernandez-Orallo, Claes Strannegard, Kristinn R. Thorisson
Place of PublicationPalo Alto CA USA
PublisherAssociation for the Advancement of Artificial Intelligence (AAAI)
Number of pages8
Publication statusPublished - 2016
EventEvaluating General Purpose AI 2016 - The Hague, Netherlands
Duration: 30 Aug 201630 Aug 2016


ConferenceEvaluating General Purpose AI 2016
Abbreviated titleEGPAI 2016
CityThe Hague
Internet address

Cite this