Assessing the effectiveness of Shah's innovation metrics for measuring innovative design within a virtual design space

Abigail Jane Kulhanek, Matthew Raymond Markovetz, Golnaz Arastoopour Irgens, Zachari Lucius Swiecki, Naomi C. Chesler, David Williamson Shaffer, Cheryl A. Bodnar

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review


Epistemic games, such as the virtual engineering internship Nephrotex, allow students to explore creative ways to approaching engineering problems while providing a novel alternative to the direct transmission method of instruction. Within Nephrotex, students choose a polymer, manufacturing process, surfactant, and percentage of carbon nanotube to create a functioning kidney dialysis membrane prototype. The performance of the membrane is measured using cost, flux, blood cell reactivity, marketability, and reliability thresholds given by stakeholders within the fictitious company. Although Nephrotex has been shown to be a valuable educational tool for modeling the product design process, only limited work has been done to investigate whether it is capable of providing an environment that allows students to generate innovative designs. The innovation assessment framework of Shah and colleagues employs four metrics of innovation - novelty, variety, quality, and quantity; novelty is further divided into a priori and a posteriori metrics. This work found that a priori and a posteriori novelty, variety, and quality were applicable metrics of innovation in the epistemic game environment of Nephrotex. Literature ranges for a priori and a posteriori novelty scores aligned with those found in this study. Comparing prior work on Nephrotex that identified innovative student designs based on a proposed literature definition, it was found that the Shah metrics between the innovative and non-innovative groups showed little variation and no statistically significant differences. A t test and a Mann-Whitney U test showed no significant difference between innovative and noninnovative groups with regard to variety or novelty scores; however, these tests did show a significant difference between groups with regards to the quality score. The same results were found when calculating Cohen's Effect Size - a priori novelty, a posteriori novelty, and variety had a small effect when comparisons were made between the innovative and non-innovative groups while quality had a large effect. The significant difference and large effect in regards to quality however, may be the result of the previous literature definition which employed quality as a measure to define innovation. Results from this study demonstrate that novelty is perhaps the most aligned innovation metric for an epistemic game environment and that both variety and quality can be helpful in understanding the designs generated within these contexts although they may need adjustment based on the application to a constrained design space.

Original languageEnglish
Title of host publication124th American Society for Engineering Education Annual Conference and Exposition 2017
Place of PublicationWashington DC USA
PublisherAmerican Society for Engineering Education
Number of pages17
ISBN (Electronic)9781510853805
Publication statusPublished - 2017
Externally publishedYes
EventASEE Annual Conference and Exposition 2017 - Columbus, United States of America
Duration: 25 Jun 201728 Jun 2017
Conference number: 124th (Website) (Proceedings)


ConferenceASEE Annual Conference and Exposition 2017
Country/TerritoryUnited States of America
Internet address

Cite this