Creedo—scalable and repeatable extrinsic evaluation for pattern discovery systems by online user studies

Mario Boley, Maike Krause-Traudes, Bo Kang, Björn Jacobs

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearch


We present Creedo|a conceptual framework along with an accompanying software implementation for empirically evaluating knowledge discovery systems. Creedo provides a
language for the creation of study designs that test how well different test systems support real users to perform certain data analysis tasks. These designs are scalable and repeatable, i.e., after their creation, a study can be carried out any number of times and with an arbitrary high number of participants without consuming valuable resources such as the time of supervision personnel. Building on the conceptual framework, the accompanying web application, which is freely available at Bitbucket, supports data mining researchers in all central tasks for conducting a user study: in embedding their ideas into an operational data analysis environment, in assigning and monitoring tasks for study
participants, and in evaluating the results. The implementation as web application enables large scale and geographically distributed studies, in which, nevertheless, all study
participants essentially have an identical user-experience.
Original languageEnglish
Title of host publicationProceedings of the ACM SIGKDD 2015 Full-day Workshop on Interactive Data Exploration and Analytics, IDEA 2015
EditorsPolo Chau, Jilles Vreeken, Matthijs van Leeuwen, Dafna Shahaf, Christos Faloutsos
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages9
Publication statusPublished - 2015
Externally publishedYes
EventACM SIGKDD Workshop on Interactive Data Exploration and Analytics 2015 - Sydney, Australia
Duration: 10 Aug 201510 Aug 2016


ConferenceACM SIGKDD Workshop on Interactive Data Exploration and Analytics 2015
Abbreviated titleIDEA 2015
Internet address

Cite this