In a silent way

communication between AI and improvising musicians beyond sound

Jon McCormack, Maria Teresa Llano Rodriguez, Toby Gifford, Matthew Yee-King, Patrick Hutchings, Mark D’Inverno

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn’t exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of the collaboration. We developed a collaborative improvising AI drummer that communicates its confidence through an emoticon-based visualisation. The AI was trained on musical performance data, as well as real-time skin conductance, of musicians improvising with professional drummers, exposing both musical and extra-musical cues to inform its generative process. Uni- and bi-directional extra-musical communication with real and false values were tested by experienced improvising musicians. Each condition was evaluated using the FSS-2 questionnaire, as a proxy for musical engagement. The results show a positive correlation between extra-musical communication of machine internal state and human musical engagement.

Original languageEnglish
Title of host publicationCHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
Subtitle of host publicationMay 4–9, 2019 Glasgow, Scotland UK
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages11
ISBN (Electronic)9781450359702
DOIs
Publication statusPublished - 2019
EventInternational Conference on Human Factors in Computing Systems 2019 - Glasgow, United Kingdom
Duration: 4 May 20199 May 2019
https://chi2019.acm.org

Conference

ConferenceInternational Conference on Human Factors in Computing Systems 2019
Abbreviated titleCHI 2019
CountryUnited Kingdom
CityGlasgow
Period4/05/199/05/19
Internet address

Keywords

  • AI systems
  • Extra-musical communication
  • Improvisation

Cite this

McCormack, J., Teresa Llano Rodriguez, M., Gifford, T., Yee-King, M., Hutchings, P., & D’Inverno, M. (2019). In a silent way: communication between AI and improvising musicians beyond sound. In CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems: May 4–9, 2019 Glasgow, Scotland UK [38] New York NY USA: Association for Computing Machinery (ACM). https://doi.org/10.1145/3290605.3300268
McCormack, Jon ; Teresa Llano Rodriguez, Maria ; Gifford, Toby ; Yee-King, Matthew ; Hutchings, Patrick ; D’Inverno, Mark. / In a silent way : communication between AI and improvising musicians beyond sound. CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems: May 4–9, 2019 Glasgow, Scotland UK. New York NY USA : Association for Computing Machinery (ACM), 2019.
@inproceedings{b7a51958007b4f128a6eb943795866bf,
title = "In a silent way: communication between AI and improvising musicians beyond sound",
abstract = "Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn’t exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of the collaboration. We developed a collaborative improvising AI drummer that communicates its confidence through an emoticon-based visualisation. The AI was trained on musical performance data, as well as real-time skin conductance, of musicians improvising with professional drummers, exposing both musical and extra-musical cues to inform its generative process. Uni- and bi-directional extra-musical communication with real and false values were tested by experienced improvising musicians. Each condition was evaluated using the FSS-2 questionnaire, as a proxy for musical engagement. The results show a positive correlation between extra-musical communication of machine internal state and human musical engagement.",
keywords = "AI systems, Extra-musical communication, Improvisation",
author = "Jon McCormack and {Teresa Llano Rodriguez}, Maria and Toby Gifford and Matthew Yee-King and Patrick Hutchings and Mark D’Inverno",
year = "2019",
doi = "10.1145/3290605.3300268",
language = "English",
booktitle = "CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems",
publisher = "Association for Computing Machinery (ACM)",
address = "United States of America",

}

McCormack, J, Teresa Llano Rodriguez, M, Gifford, T, Yee-King, M, Hutchings, P & D’Inverno, M 2019, In a silent way: communication between AI and improvising musicians beyond sound. in CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems: May 4–9, 2019 Glasgow, Scotland UK., 38, Association for Computing Machinery (ACM), New York NY USA, International Conference on Human Factors in Computing Systems 2019, Glasgow, United Kingdom, 4/05/19. https://doi.org/10.1145/3290605.3300268

In a silent way : communication between AI and improvising musicians beyond sound. / McCormack, Jon; Teresa Llano Rodriguez, Maria; Gifford, Toby; Yee-King, Matthew; Hutchings, Patrick; D’Inverno, Mark.

CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems: May 4–9, 2019 Glasgow, Scotland UK. New York NY USA : Association for Computing Machinery (ACM), 2019. 38.

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - In a silent way

T2 - communication between AI and improvising musicians beyond sound

AU - McCormack, Jon

AU - Teresa Llano Rodriguez, Maria

AU - Gifford, Toby

AU - Yee-King, Matthew

AU - Hutchings, Patrick

AU - D’Inverno, Mark

PY - 2019

Y1 - 2019

N2 - Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn’t exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of the collaboration. We developed a collaborative improvising AI drummer that communicates its confidence through an emoticon-based visualisation. The AI was trained on musical performance data, as well as real-time skin conductance, of musicians improvising with professional drummers, exposing both musical and extra-musical cues to inform its generative process. Uni- and bi-directional extra-musical communication with real and false values were tested by experienced improvising musicians. Each condition was evaluated using the FSS-2 questionnaire, as a proxy for musical engagement. The results show a positive correlation between extra-musical communication of machine internal state and human musical engagement.

AB - Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn’t exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of the collaboration. We developed a collaborative improvising AI drummer that communicates its confidence through an emoticon-based visualisation. The AI was trained on musical performance data, as well as real-time skin conductance, of musicians improvising with professional drummers, exposing both musical and extra-musical cues to inform its generative process. Uni- and bi-directional extra-musical communication with real and false values were tested by experienced improvising musicians. Each condition was evaluated using the FSS-2 questionnaire, as a proxy for musical engagement. The results show a positive correlation between extra-musical communication of machine internal state and human musical engagement.

KW - AI systems

KW - Extra-musical communication

KW - Improvisation

UR - http://www.scopus.com/inward/record.url?scp=85067600532&partnerID=8YFLogxK

U2 - 10.1145/3290605.3300268

DO - 10.1145/3290605.3300268

M3 - Conference Paper

BT - CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

PB - Association for Computing Machinery (ACM)

CY - New York NY USA

ER -

McCormack J, Teresa Llano Rodriguez M, Gifford T, Yee-King M, Hutchings P, D’Inverno M. In a silent way: communication between AI and improvising musicians beyond sound. In CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems: May 4–9, 2019 Glasgow, Scotland UK. New York NY USA: Association for Computing Machinery (ACM). 2019. 38 https://doi.org/10.1145/3290605.3300268