Epistemic network analysis as a tool for engineering design assessment

Using Epistemic Network Analysis as a Tool for Measuring Engineering Design ThinkingDesign is a central activity in engineering [1], [2]. Engineers identify problems and use theirtechnical knowledge to design a solution for a client. Doing so is an iterative process ofquantified choice under constraint. To prepare future generations of engineers to be innovativedesigners, educators and researchers must fundamentally understand the complex nature ofdesign thinking in the context of real-world engineering. In recent decades, many models fordescribing the engineering design process have been developed ranging from linear [3] toiterative [4] and from prescriptive [5] to heuristic [6]. Critical stages in the design processinclude brainstorming, generating alternative designs, conducting research [4], finding anindependent design [7], and creating ranking matrices or objective trees [8].While these models give educators rough blueprints for design activities, they do not provideinformation about the complex nature of design thinking and the way that this thinking develops.The development of design thinking requires an understanding of technical knowledge and skills,as well as the intellectual warrants that validate activity [9]. More important, it does not meanjust acquiring these problem solving elements, but about understanding the relationships amongthem. In this view, complex design thinking can be thought of in terms of connections thatengineers make as they are designing: which values to consider before taking a certain action orwhat knowledge to gather before making a particular decision. This type of connected thinkinghas been generalized to other professions as well, which Shaffer [10] has operationalized interms of an epistemic frame—a way that professionals connect skills, values, identities,knowledge, and epistemologies to solve complex problems in the field. In turn, we callconnected engineering design thinking, epistemic design thinking.To assess engineering students’ epistemic design thinking, we can use epistemic networkanalysis (ENA). Originally designed to measure the connections between elements in epistemicframes [11], [12], ENA has the capability of measuring connections in discourse and actions asthey manifest over time. Since we describe epistemic design thinking as connected learning anditeration over time, ENA may be an effective way to quantify the engineering design process asnetwork of connections and in turn, be an assessment tool for engineering design learning.In this paper, we will describe the method of ENA and provide a case study of its usage as anengineering design assessment tool. First, we will collect discourse and engineering notebookdata from an engineering virtual internship program in which students participate in solving anauthentic engineering design problem [13]. Then, we will assess design thinking by (1) applyinga traditional design rubric used in an introductory engineering course and (2) modeling networksof engineering design thinking over time using ENA. We will then correlate both conditions withthe quality of students’ design products and determine which method of assessment is a betterpredictor of high quality engineering design solutions.References[1] C. L. Dym, A. Agogino, O. Eris, D. Frey, and L. Leifer, “Engineering design thinking, teaching and learning,” J. Eng. Educ., vol. 94, no. 1, pp. 103–120, 2005.[2] H. A. Simon, The sciences of the artificial, 3rd ed. Cambridge, MA: MIT Press, 1996.[3] M. French, Form, Structure, and Mechanism. London, UK: Macmillan, 1992.[4] N. Cross, Engineering Design Methods: Strategies for Product Design, 2nd ed. Chichester, UK: John Wiley & Sons, 1994.[5] G. Pahl and W. Beitz, Engineering Design, Section Ed. London, UK: Springer-Verlag, 1996.[6] C. L. Dym and P. Little, Engineering Design: A Project-Based Introduction, 2nd ed. Hoboken, NJ: Wiley, 2003.[7] N. P. Suh, The Principles of Design. New York, NY: Oxford University Press, 1990.[8] C. L. Dym, Engineering Design A Synthesis of Views. Cambridge, MA: Cambridge University Press, 1994.[9] D. W. Shaffer, “Learning in Design,” in Foundations for the Future in Mathematics Education, R. A. Lesh, J. J. Kaput, and E. Hamilton, Eds. Mahweh, NJ: Lawrence Erlbaum Associates, 2007, pp. 99–126.[10] D. W. Shaffer, How Computer Games Help Children Learn. New York: Palgrave, 2007.[11] G. Arastoopour, N. C. Chesler, and D. W. Shaffer, “Epistemic persistence: A simulation- based approach to increasing participation of women in engineering,” J. Women Minor. Sci. Eng., vol. 20, no. 3, pp. 211–234, 2014.[12] D. W. Shaffer, D. Hatfield, G. Svarovsky, P. Nash, A. Nulty, E. A. Bagley, K. Franke, A. Rupp, and R. Mislevy, “Epistemic Network Analysis: A prototype for 21st century assessement of learning,” Int. J. Learn. Media, vol. 1, no. 1, pp. 1–21, 2009.[13] N. C. Chesler, G. Arastoopour, C. M. D’Angelo, E. A. Bagley, and D. W. Shaffer, “Design of professional practice simulator for educating and motivating first-year engineering students.,” Adv. Eng. Educ., 2012.