Creedo—scalable and repeatable extrinsic evaluation for pattern discovery systems by online user studies

Mario Boley; Maike Krause-Traudes; Bo Kang; Björn  Jacobs

Creedo—scalable and repeatable extrinsic evaluation for pattern discovery systems by online user studies

Mario Boley
, Maike Krause-Traudes
, Bo Kang
, Björn Jacobs

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research

Abstract

We present Creedo|a conceptual framework along with an accompanying software implementation for empirically evaluating knowledge discovery systems. Creedo provides a
language for the creation of study designs that test how well different test systems support real users to perform certain data analysis tasks. These designs are scalable and repeatable, i.e., after their creation, a study can be carried out any number of times and with an arbitrary high number of participants without consuming valuable resources such as the time of supervision personnel. Building on the conceptual framework, the accompanying web application, which is freely available at Bitbucket, supports data mining researchers in all central tasks for conducting a user study: in embedding their ideas into an operational data analysis environment, in assigning and monitoring tasks for study
participants, and in evaluating the results. The implementation as web application enables large scale and geographically distributed studies, in which, nevertheless, all study
participants essentially have an identical user-experience.

Original language	English
Title of host publication	Proceedings of the ACM SIGKDD 2015 Full-day Workshop on Interactive Data Exploration and Analytics, IDEA 2015
Editors	Polo Chau, Jilles Vreeken, Matthijs van Leeuwen, Dafna Shahaf, Christos Faloutsos
Place of Publication	New York NY USA
Publisher	Association for Computing Machinery (ACM)
Pages	20-28
Number of pages	9
Publication status	Published - 2015
Externally published	Yes
Event	ACM SIGKDD Workshop on Interactive Data Exploration and Analytics 2015 - Sydney, Australia Duration: 10 Aug 2015 → 10 Aug 2016 http://poloclub.gatech.edu/idea2015/index.html

Conference

Conference	ACM SIGKDD Workshop on Interactive Data Exploration and Analytics 2015
Abbreviated title	IDEA 2015
Country/Territory	Australia
City	Sydney
Period	10/08/15 → 10/08/16
Internet address	http://poloclub.gatech.edu/idea2015/index.html

Access to Document

280354635-oaFinal published version, 1.36 MB

http://www.realkd.org/wp-content/uploads/2015/08/boley2015creedo.pdf

Fraunhofer Institut Autonome Intelligente Systeme (Fraunhofer Institute for Intelligent Analysis and Information Systems) (External organisation)
Boley, M. (Inventor) & Wrobel, S. (Inventor)
4 Apr 2014 → 4 Apr 2034
Activity: Industry, Government and Philanthropy Engagement and Partnerships › Patents › Patent

Cite this

Boley, M., Krause-Traudes, M., Kang, B., & Jacobs, B. (2015). Creedo—scalable and repeatable extrinsic evaluation for pattern discovery systems by online user studies. In P. Chau, J. Vreeken, M. van Leeuwen, D. Shahaf, & C. Faloutsos (Eds.), Proceedings of the ACM SIGKDD 2015 Full-day Workshop on Interactive Data Exploration and Analytics, IDEA 2015 (pp. 20-28). Association for Computing Machinery (ACM). http://www.realkd.org/wp-content/uploads/2015/08/boley2015creedo.pdf

Boley, Mario ; Krause-Traudes, Maike ; Kang, Bo et al. / Creedo—scalable and repeatable extrinsic evaluation for pattern discovery systems by online user studies. Proceedings of the ACM SIGKDD 2015 Full-day Workshop on Interactive Data Exploration and Analytics, IDEA 2015. editor / Polo Chau ; Jilles Vreeken ; Matthijs van Leeuwen ; Dafna Shahaf ; Christos Faloutsos. New York NY USA : Association for Computing Machinery (ACM), 2015. pp. 20-28

@inproceedings{ddcc0ec30ae64d4faf97457ba89c780a,

title = "Creedo—scalable and repeatable extrinsic evaluation for pattern discovery systems by online user studies",

abstract = "We present Creedo|a conceptual framework along with an accompanying software implementation for empirically evaluating knowledge discovery systems. Creedo provides a language for the creation of study designs that test how well different test systems support real users to perform certain data analysis tasks. These designs are scalable and repeatable, i.e., after their creation, a study can be carried out any number of times and with an arbitrary high number of participants without consuming valuable resources such as the time of supervision personnel. Building on the conceptual framework, the accompanying web application, which is freely available at Bitbucket, supports data mining researchers in all central tasks for conducting a user study: in embedding their ideas into an operational data analysis environment, in assigning and monitoring tasks for study participants, and in evaluating the results. The implementation as web application enables large scale and geographically distributed studies, in which, nevertheless, all study participants essentially have an identical user-experience.",

author = "Mario Boley and Maike Krause-Traudes and Bo Kang and Bj{\"o}rn Jacobs",

year = "2015",

language = "English",

pages = "20--28",

editor = "Polo Chau and Jilles Vreeken and \{van Leeuwen\}, Matthijs and Dafna Shahaf and Faloutsos, \{Christos \}",

booktitle = "Proceedings of the ACM SIGKDD 2015 Full-day Workshop on Interactive Data Exploration and Analytics, IDEA 2015",

publisher = "Association for Computing Machinery (ACM)",

address = "United States of America",

note = "ACM SIGKDD Workshop on Interactive Data Exploration and Analytics 2015, IDEA 2015 ; Conference date: 10-08-2015 Through 10-08-2016",

url = "http://poloclub.gatech.edu/idea2015/index.html",

}

Boley, M, Krause-Traudes, M, Kang, B & Jacobs, B 2015, Creedo—scalable and repeatable extrinsic evaluation for pattern discovery systems by online user studies. in P Chau, J Vreeken, M van Leeuwen, D Shahaf & C Faloutsos (eds), Proceedings of the ACM SIGKDD 2015 Full-day Workshop on Interactive Data Exploration and Analytics, IDEA 2015. Association for Computing Machinery (ACM), New York NY USA, pp. 20-28, ACM SIGKDD Workshop on Interactive Data Exploration and Analytics 2015, Sydney, New South Wales, Australia, 10/08/15. <http://www.realkd.org/wp-content/uploads/2015/08/boley2015creedo.pdf>

Creedo—scalable and repeatable extrinsic evaluation for pattern discovery systems by online user studies. / Boley, Mario; Krause-Traudes, Maike; Kang, Bo et al.
Proceedings of the ACM SIGKDD 2015 Full-day Workshop on Interactive Data Exploration and Analytics, IDEA 2015. ed. / Polo Chau; Jilles Vreeken; Matthijs van Leeuwen; Dafna Shahaf; Christos Faloutsos. New York NY USA: Association for Computing Machinery (ACM), 2015. p. 20-28.

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research

TY - GEN

T1 - Creedo—scalable and repeatable extrinsic evaluation for pattern discovery systems by online user studies

AU - Boley, Mario

AU - Krause-Traudes, Maike

AU - Kang, Bo

AU - Jacobs, Björn

PY - 2015

Y1 - 2015

N2 - We present Creedo|a conceptual framework along with an accompanying software implementation for empirically evaluating knowledge discovery systems. Creedo provides a language for the creation of study designs that test how well different test systems support real users to perform certain data analysis tasks. These designs are scalable and repeatable, i.e., after their creation, a study can be carried out any number of times and with an arbitrary high number of participants without consuming valuable resources such as the time of supervision personnel. Building on the conceptual framework, the accompanying web application, which is freely available at Bitbucket, supports data mining researchers in all central tasks for conducting a user study: in embedding their ideas into an operational data analysis environment, in assigning and monitoring tasks for study participants, and in evaluating the results. The implementation as web application enables large scale and geographically distributed studies, in which, nevertheless, all study participants essentially have an identical user-experience.

AB - We present Creedo|a conceptual framework along with an accompanying software implementation for empirically evaluating knowledge discovery systems. Creedo provides a language for the creation of study designs that test how well different test systems support real users to perform certain data analysis tasks. These designs are scalable and repeatable, i.e., after their creation, a study can be carried out any number of times and with an arbitrary high number of participants without consuming valuable resources such as the time of supervision personnel. Building on the conceptual framework, the accompanying web application, which is freely available at Bitbucket, supports data mining researchers in all central tasks for conducting a user study: in embedding their ideas into an operational data analysis environment, in assigning and monitoring tasks for study participants, and in evaluating the results. The implementation as web application enables large scale and geographically distributed studies, in which, nevertheless, all study participants essentially have an identical user-experience.

M3 - Conference Paper

SP - 20

EP - 28

BT - Proceedings of the ACM SIGKDD 2015 Full-day Workshop on Interactive Data Exploration and Analytics, IDEA 2015

A2 - Chau, Polo

A2 - Vreeken, Jilles

A2 - van Leeuwen, Matthijs

A2 - Shahaf, Dafna

A2 - Faloutsos, Christos

PB - Association for Computing Machinery (ACM)

CY - New York NY USA

T2 - ACM SIGKDD Workshop on Interactive Data Exploration and Analytics 2015

Y2 - 10 August 2015 through 10 August 2016

ER -

Boley M, Krause-Traudes M, Kang B, Jacobs B. Creedo—scalable and repeatable extrinsic evaluation for pattern discovery systems by online user studies. In Chau P, Vreeken J, van Leeuwen M, Shahaf D, Faloutsos C, editors, Proceedings of the ACM SIGKDD 2015 Full-day Workshop on Interactive Data Exploration and Analytics, IDEA 2015. New York NY USA: Association for Computing Machinery (ACM). 2015. p. 20-28

Creedo—scalable and repeatable extrinsic evaluation for pattern discovery systems by online user studies

Abstract

Conference

Access to Document

Activities

Fraunhofer Institut Autonome Intelligente Systeme (Fraunhofer Institute for Intelligent Analysis and Information Systems) (External organisation)

Cite this