ETD
237 Pages

ETD

Course Number: LIB 04072004, Fall 2008

College/University: Virginia Tech

Word Count: 88866

Rating:

Document Preview

Tools to Improve the Process of Engineering Design: An Analysis of Team Configuration and Project Support by Paige E. Smith Dissertation submitted to the Faculty of Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Industrial and Systems Engineering Dr. Brian M. Kleiner (Chair) Dr. Joe Meredith, Jr. Dr. Maury Nussbaum Dr. Eileen...

Unformatted Document Excerpt
Coursehero >> Virginia >> Virginia Tech >> LIB 04072004

Course Hero has millions of student submitted documents similar to the one
below including study guides, practice problems, reference materials, practice exams, textbook help and tutor support.

Course Hero has millions of student submitted documents similar to the one below including study guides, practice problems, reference materials, practice exams, textbook help and tutor support.

to Tools Improve the Process of Engineering Design: An Analysis of Team Configuration and Project Support by Paige E. Smith Dissertation submitted to the Faculty of Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Industrial and Systems Engineering Dr. Brian M. Kleiner (Chair) Dr. Joe Meredith, Jr. Dr. Maury Nussbaum Dr. Eileen Van Aken Dr. Bevlee A. Watford March 24, 2004 Blacksburg, Virginia Keywords: Engineering Design, Project Management, Teams, Macroergonomics Tools to Improve the Process of Engineering Design: An Analysis of Team Configuration and Project Support Paige E. Smith Abstract The purpose of this research was to determine how team design and project management (planning and tracking) affected planning and design performance and the people involved in the process. A laboratory study was conducted to evaluate three factors: team design (individuals versus groups of three), project support (no project support versus manual project support versus automated project support), and the engineering design life-cycle, which includes conceptual design, preliminary design, and detailed design. There were six observations per treatment, involving a total of 72 undergraduate engineering students. The impact of these factors were evaluated for planning time, design cycle time, cost effectiveness, cost variance, schedule variance, mental workload, and job satisfaction. For treatments that called for groups, group process was evaluated in addition to group workload. The results showed groups took 61% more time to plan their projects compared to individuals (p<0.01). Planning time was 31% longer for participants with manual support compared to those with automated project support (p<0.01). Schedule variance (p<0.01) and cost variance (p<0.001) decreased 24% and 23%, respectively, over time during the design process. The design cycle time was 17% longer for participants without project support compared to those with automated project support (p<0.05). During design, groups and individuals allocated their time differently (p<0.05). Mental workload, measured with the NASA Task Load Index (TLX), showed workload increased 16% over time (p<0.001). In addition, the combination of design phase and project tracking support affected the TLX (p<0.01). Job satisfaction was 5% lower at the end of the design project compared to the beginning of design (p<0.05). From the analysis on group process, the type of project support affected the group process during planning. Groups with manual support interacted 83% more than those with automated support (effective behaviors: p<0.01; ineffective behaviors: p<0.05). During design, the roles individuals played within the group affected how much they contributed to the groups process (effective behaviors: p<0.0001; ineffective behaviors: p<0.01). There were several practical implications that can be drawn from this study. In the decision to use teams versus groups, there was evidence that groups were able to attend to more of the design requirements than individuals, which resulted in the design of systems with higher reliability. However the tradeoff of using groups were in the labor cost and in longer planning and status report meetings. Therefore the organizations goals need to be carefully considered before selecting the team design. For project support, there were clear benefits to automating the planning process. Automation resulted in better Gantt chart and planning sessions that were completed more quickly compared to those with manual support. Furthermore, systems designed with automated support resulted in lower design costs compared to systems designed without project support. Dedication For Mom and Dad iii Acknowledgements This dissertation was possible due to the contributions of so many people. I particularly want to thank: Dr. Brian Kleiner, my advisor, for his guidance and support throughout this endeavor and for his encouragement to seek my current position at the University of Maryland, a decision that extended my time to graduation, but which continues to be a challenging and rewarding experience. My four committee members for their encouragement and guidance: Dr. Joe Meredith whose mentorship, stories, and help with the engineering design task was invaluable. Dr. Maury Nussbaum for challenging me on the statistics and mental workload. Dr. Eileen Van Aken for advice, encouragement and management systems perspective. And a special thanks to Dr. Bevlee Watford. This all started because of you. Steven Van Aken for his help with the project management training I used in this research. To my University of Maryland colleagues: Dr. Linda Schmidt for being my University of Maryland Advisor, mentor, and role model whose constant encouragement made me realize that every day I was a little bit closer to finishing. Dr. Larry Douglass for statistics advice and teaching me a new statistical method for analyzing mixed factor designs. Dr. Janet Schmidt for locating a laboratory for my research and Dr. David Bigio for helping recruit participants. The Office of Student Affairs in the Clark School of Engineering for their support and interest and my graduate assistants who gave me constant support, encouragement, and motivation. The Clark School of Engineering students who volunteered to participate in this research. And the students at Virginia Tech in the Office of Minority Engineering Programs (from 1998-2001) for helping me pilot this study. And last but not certainly not least, a big thank you to Mom, Dad, Scott, Granny, Brenda and Scotty, for their patience, encouragement, belief in me and amazing support. iv Table of Contents ABSTRACT................................................................................................................................................II DEDICATION.......................................................................................................................................... III ACKNOWLEDGEMENTS .....................................................................................................................IV TABLE OF CONTENTS .......................................................................................................................... V LIST OF TABLES ................................................................................................................................ VIII LIST OF FIGURES ...............................................................................................................................XIV CHAPTER 1 1.1 1.2 1.3 1.4 INTRODUCTION ........................................................................................................ 1 PROBLEM STATEMENT................................................................................................................. 2 RESEARCH OBJECTIVES ............................................................................................................... 2 QUESTIONS AND HYPOTHESES .................................................................................................... 2 RESEARCH OVERVIEW................................................................................................................. 3 LITERATURE REVIEW ............................................................................................ 5 CHAPTER 2 2.1 THE MACROERGONOMIC PERSPECTIVE....................................................................................... 5 2.1.1 Sociotechnical Systems Theory ............................................................................................... 5 2.1.2 Function Allocation................................................................................................................. 7 2.2 PERSONNEL SUBSYSTEM ............................................................................................................. 7 2.2.1 Workload ................................................................................................................................. 8 2.3 TECHNOLOGICAL SUBSYSTEM..................................................................................................... 8 2.3.1 Engineering Analysis and Design ........................................................................................... 9 2.3.2 Project Management ............................................................................................................. 11 2.4 FUNCTION ALLOCATION REVISITED.......................................................................................... 12 2.5 EXTERNAL ENVIRONMENT ........................................................................................................ 14 2.6 ORGANIZATIONAL DESIGN ........................................................................................................ 14 2.6.1 Engineering Design Teams ................................................................................................... 15 2.6.2 Project Teams........................................................................................................................ 17 2.6.3 Groups versus Individuals..................................................................................................... 17 2.6.4 Group Size............................................................................................................................. 19 2.6.5 Group Process....................................................................................................................... 19 2.6.6 Job Satisfaction ..................................................................................................................... 25 2.7 PERFORMANCE MEASURES........................................................................................................ 26 2.7.1 Technological Subsystem Performance Measures ................................................................ 26 2.7.2 Personnel Subsystem Performance Measures....................................................................... 27 2.7.3 Job Satisfaction ..................................................................................................................... 31 2.7.4 Group Process....................................................................................................................... 32 CHAPTER 3 EXPERIMENTAL DESIGN ..................................................................................... 34 3.1 SUBJECTS ................................................................................................................................... 36 3.2 MATERIALS AND EQUIPMENT.................................................................................................... 38 3.2.1 Engineering Design Project .................................................................................................. 38 3.2.2 Supplemental Forms.............................................................................................................. 39 v 3.2.3 Script ..................................................................................................................................... 40 3.3 FACILITY.................................................................................................................................... 40 3.4 PROCEDURE ............................................................................................................................... 40 3.4.1 Pilot Testing .......................................................................................................................... 40 3.4.2 Experimental Procedure ....................................................................................................... 41 3.5 DEPENDENT VARIABLES............................................................................................................ 43 3.5.1 Design Performance.............................................................................................................. 43 3.5.2 Planning and Tracking Performance .................................................................................... 44 3.5.3 Mental Workload................................................................................................................... 44 3.5.4 Job Satisfaction ..................................................................................................................... 45 3.5.5 Group Process....................................................................................................................... 45 3.6 DATA ANALYSIS ........................................................................................................................ 45 3.7 ASSUMPTIONS AND LIMITATIONS.............................................................................................. 47 CHAPTER 4 RESULTS FOR FIRST ORDER VARIABLES ...................................................... 49 4.1 PARTICIPANT DEMOGRAPHICS .................................................................................................. 49 4.2 DESCRIPTIVE STATISTICS .......................................................................................................... 52 4.3 EXPLORING ANOVA ASSUMPTIONS ......................................................................................... 54 4.4 PERFORMANCE........................................................................................................................... 55 4.4.1 Design Performance.............................................................................................................. 55 4.4.2 Planning Performance .......................................................................................................... 58 4.5 NASA TLX................................................................................................................................ 60 4.6 JOB SATISFACTION..................................................................................................................... 62 CHAPTER 5 RESULTS FOR SECOND ORDER VARIABLES ................................................. 64 5.1 INTRODUCTION .......................................................................................................................... 64 5.2 EXPLORING ANOVA ASSUMPTIONS ......................................................................................... 64 5.3 PERFORMANCE........................................................................................................................... 67 5.3.1 Design Performance.............................................................................................................. 67 5.3.2 Planning Performance .......................................................................................................... 70 5.4 NASA TLX................................................................................................................................ 70 5.5 JOB SATISFACTION..................................................................................................................... 76 5.6 SUPPLEMENTAL QUESTIONNAIRE RESPONSES .......................................................................... 87 5.6.1 Design Related Questions ..................................................................................................... 87 5.6.2 Planning and Tracking Related Questions............................................................................ 88 5.7 CORRELATIONS.......................................................................................................................... 90 CHAPTER 6 RESULTS FOR ROLE DATA.................................................................................. 92 6.1 NASA TLX................................................................................................................................ 98 6.1.1 Analysis of the NASA TLX for Groups by Role ..................................................................... 98 6.1.2 NASA TLX in Planning.......................................................................................................... 98 6.1.3 NASA TLX in the Design Process ......................................................................................... 99 6.1.4 Reflective NASA TLX........................................................................................................... 105 6.2 JOB SATISFACTION................................................................................................................... 105 6.3 SUPPLEMENTAL DESIGN QUESTIONS....................................................................................... 123 6.4 SUPPLEMENTAL PLANNING QUESTIONS .................................................................................. 124 6.5 GROUP WORKLOAD ................................................................................................................. 129 6.6 GROUP WORKLOAD EVALUATED BY OUTSIDER OBSERVERS ................................................. 132 6.7 CRITICAL TEAM BEHAVIORS ................................................................................................... 134 6.7.1 Exploring Effective and Ineffective Behaviors .................................................................... 134 6.7.2 Critical Team Behaviors in Planning.................................................................................. 136 vi 6.7.3 Critical Team Behaviors in the Design Process ................................................................. 138 6.7.4 Correlations with Critical Team Behaviors ........................................................................ 146 6.8 SUPPLEMENTAL GROUP PROCESS OBSERVATIONS ................................................................. 150 6.9 CORRELATIONS........................................................................................................................ 156 CHAPTER 7 DISCUSSION............................................................................................................ 158 7.1 PERFORMANCE......................................................................................................................... 158 7.1.1 Design Performance............................................................................................................ 158 7.1.2 Design Cycle Time .............................................................................................................. 164 7.1.3 Planning Performance ........................................................................................................ 165 7.1.4 Planning and Tracking Time............................................................................................... 168 7.2 NASA TLX.............................................................................................................................. 168 7.2.1 NASA TLX during Planning ................................................................................................ 168 7.2.2 NASA TLX during Design ................................................................................................... 169 7.2.3 Reflective NASA TLX........................................................................................................... 172 7.2.4 Analysis of the NASA TLX for Groups by Role ................................................................... 172 7.3 JOB SATISFACTION................................................................................................................... 175 7.3.1 Job Satisfaction during Planning........................................................................................ 175 7.3.2 Job Satisfaction during Design ........................................................................................... 176 7.3.3 Reflective Job Satisfaction .................................................................................................. 180 7.3.4 Analysis of Job Satisfaction by Role ................................................................................... 180 7.4 RELATIONSHIP BETWEEN JOB SATISFACTION AND WORKLOAD OR PERFORMANCE .............. 187 7.5 GROUP WORKLOAD ................................................................................................................. 187 7.5.1 Group Workload Evaluated by Outsider Observers ........................................................... 189 7.6 CRITICAL TEAM BEHAVIORS ................................................................................................... 190 7.6.1 Correlations with Critical Team Behaviors ........................................................................ 192 7.6.2 Supplemental Group Process Observations........................................................................ 194 CHAPTER 8 8.1 8.2 8.3 8.4 8.5 CONCLUSIONS....................................................................................................... 196 TEAM DESIGN .......................................................................................................................... 196 PROJECT SUPPORT ................................................................................................................... 197 PRELIMINARY GUIDELINES...................................................................................................... 198 FUTURE RESEARCH.................................................................................................................. 199 PRACTICAL IMPLICATIONS ...................................................................................................... 201 REFERENCES........................................................................................................................................ 203 APPENDICES ......................................................................................................................................... 223 vii List of Tables Table 2.1 Taxonomy for the classification of automation levels and operator roles augmented with levels of workload ........................................................................................................................................ 12 Table 3.1 Design matrix for experiment ..................................................................................................... 34 Table 3.2 Design matrix for the 2x2 factorial between subjects design ..................................................... 35 Table 3.3 Design matrix for the 2x2x3 mixed factor design....................................................................... 35 Table 3.4 Number of replicates required for detecting a difference in cost effectiveness .......................... 36 Table 3.5 Number of replicates required for detecting a difference in number of ideas............................. 37 Table 3.6 Number of replicates required for detecting a difference in the NASA TLX............................. 37 Table 3.7 Number of replicates to detect a difference in overall team workload ....................................... 37 Table 3.8 Number of replicates required for detecting a difference in overall team workload .................. 37 Table 3.9 Components of cost effectiveness............................................................................................... 44 Table 4.1 Gender composition in treatments .............................................................................................. 50 Table 4.2 Mean cost effectiveness for each treatment ................................................................................ 52 Table 4.3 Mean life-cycle cost for each treatment...................................................................................... 53 Table 4.4 Mean material costs and standard deviations for each condition................................................ 53 Table 4.5 Mean design cost in each treatment group.................................................................................. 53 Table 4.6 Mean system effectiveness in each treatment group................................................................... 53 Table 4.7 Mean reliability and standard deviations for each treatment group ............................................ 53 Table 4.8 Summary of data analysis1 .......................................................................................................... 55 Table 4.9 ANOVA for design cycle time ................................................................................................... 56 Table 4.10 Comparisons of mean design cycle time for type of project support........................................ 56 Table 4.11 Variance analysis conducted for time in each design phase1 .................................................... 57 Table 4.12 Comparisons of the mean time in each design phase................................................................ 57 Table 4.13 Comparisons of mean design time based on design phase and team design............................. 57 Table 4.14 ANOVA for planning time ....................................................................................................... 58 Table 4.15 Variance analysis for status report meeting times .................................................................... 59 Table 4.16 Multiple comparisons of mean status meeting time based on report period............................. 59 Table 4.17 Variance analysis for the cost performance index (CPI) .......................................................... 59 Table 4.18 Multiple comparisons of the mean CPI based on report period................................................ 60 Table 4.19 Variance analysis for the schedule performance index (SPI) ................................................... 60 Table 4.20 Multiple comparisons of the mean SPI for each reporting period ............................................ 60 Table 4.21 Analysis of the NASA TLX during design............................................................................... 61 Table 4.22 Multiple comparisons of the mean NASA TLX in each design phase ..................................... 61 Table 4.23 Comparisons of NASA TLX for the interaction between design phase and project support ... 61 Table 4.24 Variance analysis for job satisfaction during design ................................................................ 63 Table 4.25 Comparisons of the mean job satisfaction in each design phase .............................................. 63 Table 5.1 Summary of the data analysis for supplemental performance variables..................................... 64 Table 5.2 Summary of the data analysis for supplemental variables .......................................................... 65 Table 5.3 MANOVA to test the affect of reliability, robustness, system size, and producibility on main effects and interactions....................................................................................................................... 67 Table 5.4 ANOVA for reliability................................................................................................................ 68 Table 5.5 MANOVA to test the affect of life-cycle, design, and material costs on main effects and interactions ......................................................................................................................................... 68 Table 5.6 ANOVA for life-cycle cost......................................................................................................... 68 Table 5.7 Variance analysis for design cost................................................................................................ 69 Table 5.8 Comparisons of mean design cost based on type of project support .......................................... 69 viii Table 5.9 MANOVA to test the affect of total, moving and unique parts on main effects and interactions ............................................................................................................................................................ 69 Table 5.10 ANOVA for number of moving parts ....................................................................................... 69 Table 5.11 MANOVA to test the affect of concepts and design criteria on main effects and interactions 69 Table 5.12 MANOVA to test the affect of planning documents on main effects and interactions ............ 70 Table 5.13 ANOVA for the Gantt chart score ............................................................................................ 70 Table 5.14 MANOVA for NASA TLX components from planning .......................................................... 70 Table 5.15 MANOVA for NASA TLX components during design ........................................................... 71 Table 5.16 Analysis of mental demand during design................................................................................ 71 Table 5.17 Comparisons of mental demand for the interaction between design phase and team design ... 71 Table 5.18 Variance analysis on physical demand during design .............................................................. 72 Table 5.19 Multiple comparisons of physical demand in each design phase ............................................. 72 Table 5.20 Variance analysis for temporal demand during design ............................................................. 73 Table 5.21 Multiple comparisons of temporal demand in each design phase............................................. 73 Table 5.22 Variance analysis on effort during design................................................................................. 74 Table 5.23 Multiple comparisons of effort in each design phase ............................................................... 74 Table 5.24 Comparisons of effort for the interaction between design phase and project support1 ............. 74 Table 5.25 Variance analysis for frustration during design ........................................................................ 75 Table 5.26 Multiple comparisons of frustration in each design phase........................................................ 75 Table 5.27 MANOVA to test the affect of reflective NASA TLX components......................................... 76 Table 5.28 ANOVA for reflective frustration............................................................................................. 76 Table 5.29 MANOVA to test the affect of job satisfaction components on main effects and interactions during planning .................................................................................................................................. 76 Table 5.30 MANOVA for responses to questions used to determine job satisfaction during planning ..... 76 Table 5.31 ANOVA for the perception of enough time for planning ......................................................... 77 Table 5.32 MANOVA to test the affect of comfort, challenge, and resources during design .................... 77 Table 5.33 MANOVA on responses to questions used to determine job satisfaction during design ......... 78 Table 5.34 ANOVA of comfort during design ........................................................................................... 78 Table 5.35 Multiple comparisons of comfort in each design phase............................................................ 78 Table 5.36 Analysis of physical surroundings during design ..................................................................... 79 Table 5.37 Comparisons of physical surroundings based on team design and project support .................. 79 Table 5.38 Analysis of perceived time during design................................................................................. 80 Table 5.39 Comparisons of perceived time in each design phase............................................................... 80 Table 5.40 Analysis of excessive work during design................................................................................ 80 Table 5.41 Multiple comparisons of excessive work in each design phase................................................ 81 Table 5.42 Variance analysis of ability during the design process ............................................................. 81 Table 5.43 Multiple comparisons of ability for the interaction between team design and project support 81 Table 5.44 Variance analysis on task interest during design ...................................................................... 82 Table 5.45 Multiple comparisons of the mean task interest in each design phase...................................... 82 Table 5.46 Analysis of freedom during the design process ........................................................................ 83 Table 5.47 Multiple comparisons of freedom in each design phase ........................................................... 83 Table 5.48 Variance analysis on problem difficulty during design ............................................................ 83 Table 5.49 Comparisons of problem difficulty between each design phase............................................... 84 Table 5.50 Analysis of ability to see results during the design process...................................................... 84 Table 5.51 Multiple comparisons of the mean ability to see results based on phase.................................. 84 Table 5.52 Variance analysis on access to right equipment during design................................................. 85 Table 5.53 MANOVA to test the affect of reflective job satisfaction components on main effects and interactions ......................................................................................................................................... 85 Table 5.54 MANOVA to test the affect of responses to all reflective questions on main effects and interactions ......................................................................................................................................... 85 Table 5.55 ANOVA for reflective developing ability ................................................................................ 86 ix Table 5.56 Comparisons of reflective ability based on team design and project support........................... 86 Table 5.57 Variance analysis on doubt during design ................................................................................ 87 Table 5.58 Multiple comparisons of doubt in each design phase ............................................................... 87 Table 5.59 MANOVA to test the affect of supplemental design questions on main effects and interactions ............................................................................................................................................................ 88 Table 5.60 MANOVA to test the affect of supplemental planning questions ............................................ 88 Table 5.61 ANOVA for doubt in ability to satisfy the plan........................................................................ 88 Table 5.62 Comparisons of doubt for the interaction between team design and project support ............... 88 Table 5.63 MANOVA to test the affect of responses to supplemental questions on tracking tools ........... 89 Table 5.64 ANOVA for the project support tools ease of use of during design........................................ 89 Table 5.65 Multiple comparisons for ease of use in each design phase...................................................... 89 Table 5.66 MANOVA to test the affect of reflective supplemental planning and tracking questions ....... 90 Table 5.67 Correlation between job satisfaction/challenge and mental workload/frustration/TLX ........... 90 Table 5.68 Correlation between job satisfaction and performance............................................................. 91 Table 6.1 Summary of data analysis amongst group members for variables during planning ................... 92 Table 6.2 Summary of data analysis amongst group members for variables during design....................... 94 Table 6.3 Summary of the data analysis amongst group members for reflective variables........................ 97 Table 6.4 Reliability between the TLX for each group member and the mean TLX ................................. 98 Table 6.5 MANOVA to test the affect of NASA TLX components during planning ................................ 98 Table 6.6 ANOVA for temporal demand during planning ......................................................................... 98 Table 6.7 ANOVA for effort during planning ............................................................................................ 99 Table 6.8 Variance analysis for the NASA TLX ........................................................................................ 99 Table 6.9 Multiple comparisons of the NASA TLX in each design phase................................................. 99 Table 6.10 Comparisons of the NASA TLX based on design phase and project support ........................ 100 Table 6.11 MANOVA to test the affect of NASA TLX components on main effects and interactions... 100 Table 6.12 Variance analysis for mental demand ..................................................................................... 101 Table 6.13 Multiple comparisons of mental demand in each design phase.............................................. 101 Table 6.14 Variance analysis for physical demand................................................................................... 101 Table 6.15 Multiple comparisons of physical demand in each design phase ........................................... 102 Table 6.16 Variance analysis for temporal demand.................................................................................. 102 Table 6.17 Multiple comparisons of temporal demand in each design phase........................................... 102 Table 6.18 Variance analysis for performance ratings.............................................................................. 103 Table 6.19 Comparison of performance ratings for design phase and project support............................. 103 Table 6.20 Variance analysis for effort ratings......................................................................................... 104 Table 6.21 Multiple comparisons of effort in each design phase ............................................................. 104 Table 6.22 Variance analysis for frustration ............................................................................................. 104 Table 6.23 Multiple comparisons of frustration in each design phase...................................................... 105 Table 6.24 MANOVA to test the affect of reflective TLX components on main effects and interactions .......................................................................................................................................................... 105 Table 6.25 ANOVA for the reflective physical demand........................................................................... 105 Table 6.26 Multiple comparisons between types of project support for reflective physical demand ....... 105 Table 6.27 MANOVA to test the affect of job satisfaction components during planning........................ 106 Table 6.28 MANOVA to test the affect of responses to job satisfaction questions on main effects and interactions during planning............................................................................................................. 106 Table 6.29 ANOVA for perceived time during planning ......................................................................... 106 Table 6.30 ANOVA for challenge during planning.................................................................................. 106 Table 6.31 Multiple comparisons of challenge between project support and role.................................... 107 Table 6.32 Variance analysis for job satisfaction ..................................................................................... 108 Table 6.33 Multiple comparisons of job satisfaction in each design phase .............................................. 108 Table 6.34 Multiple comparisons of job satisfaction for each project support level ................................ 108 Table 6.35 MANOVA to test the affect of job satisfaction facets on main effects and interactions ........ 108 x Table 6.36 MANOVA to test the affect of job satisfaction responses on main effects and interactions .. 109 Table 6.37 Variance analysis for comfort ................................................................................................. 109 Table 6.38 Multiple comparisons of comfort in each design phase.......................................................... 110 Table 6.39 Multiple comparisons of comfort for project support levels................................................... 110 Table 6.40 Variance analysis for excessive work in groups ..................................................................... 110 Table 6.41 Multiple comparisons of excessive work in each design phase.............................................. 111 Table 6.42 Multiple comparisons of excessive work in each project support level.................................. 111 Table 6.43 Variance analysis for physical surroundings .......................................................................... 111 Table 6.44 Comparisons of physical surroundings based on project support........................................... 111 Table 6.45 Variance analysis for perceived time ...................................................................................... 112 Table 6.46 Multiple comparisons of perceived time in each design phase............................................... 112 Table 6.47 Variance analysis for developing ability................................................................................. 112 Table 6.48 Multiple comparisons of developing ability based on project support ................................... 113 Table 6.49 Variance analysis for interest perceptions .............................................................................. 113 Table 6.50 Multiple comparisons of perceived interest in each design phase .......................................... 113 Table 6.51 Multiple comparisons of interest for the interaction between project support and role.......... 114 Table 6.52 Variance analysis for ability to see results.............................................................................. 114 Table 6.53 Multiple comparisons of ability to see results in each design phase....................................... 115 Table 6.54 Variance analysis for resources .............................................................................................. 115 Table 6.55 Multiple comparisons of resources based on project support and role ................................... 116 Table 6.56 Variance analysis for group member competence .................................................................. 117 Table 6.57 Multiple comparisons of group member competence based on project support..................... 117 Table 6.58 Multiple comparisons of group member competence based on project support and role....... 117 Table 6.59 Variance analysis for group member helpfulness ................................................................... 118 Table 6.60 Multiple comparisons of group member helpfulness in each design phase............................ 119 Table 6.61 Multiple comparisons of helpful group members based on project support and role ............. 119 Table 6.62 MANOVA to test affect of reflective job satisfaction facets on main effects and interactions .......................................................................................................................................................... 120 Table 6.63 MANOVA to test the affect of reflective job satisfaction and supplemental design questions .......................................................................................................................................................... 120 Table 6.64 Variance analysis for reflective developing ability................................................................. 120 Table 6.65 Multiple comparisons of reflective developing ability based on project support ................... 120 Table 6.66 Variance analysis for reflective interest in the problem.......................................................... 121 Table 6.67 Comparisons of reflective interest in the problem based on project support and role ............ 121 Table 6.68 Variance analysis for reflective problem difficulty ................................................................ 122 Table 6.69 Multiple comparisons of reflective problem difficulty based on project support and role ..... 122 Table 6.70 Variance analysis for doubt .................................................................................................... 123 Table 6.71 Multiple comparisons of doubt in each design phase ............................................................. 123 Table 6.72 Multiple comparisons of mean doubt based on project support ............................................. 123 Table 6.73 ANOVA for doubt in ability during planning......................................................................... 124 Table 6.74 ANOVA for the question, creating the best plan within their ability ..................................... 124 Table 6.75 ANOVA for the ease of use during planning.......................................................................... 124 Table 6.76 ANOVA for efficiency during planning ................................................................................. 125 Table 6.77 ANOVA for productivity during planning.............................................................................. 125 Table 6.78 Multiple comparisons between roles for productivity during planning .................................. 125 Table 6.79 ANOVA for satisfaction during planning in groups ............................................................... 125 Table 6.80 MANOVA to test the affect of supplemental planning questions on main effects and interactions ....................................................................................................................................... 126 Table 6.81 Variance analysis for ease of use ............................................................................................ 126 Table 6.82 Multiple comparisons of ease of use in each design phase..................................................... 126 Table 6.83 Multiple comparisons of ease of use based on role ................................................................ 126 xi Table 6.84 Variance analysis for effectiveness......................................................................................... 127 Table 6.85 Multiple comparisons of the mean effectiveness in each design phase .................................. 127 Table 6.86 Variance analysis for satisfaction ........................................................................................... 127 Table 6.87 MANOVA to test the affect of the reflective supplemental planning questions .................... 128 Table 6.88 ANOVA for reflective ease of use.......................................................................................... 128 Table 6.89 ANOVA on reflective productivity......................................................................................... 128 Table 6.90 ANOVA on reflective satisfaction.......................................................................................... 128 Table 6.91 MANOVA to test the affect of group workload scales on main effects and interactions during planning............................................................................................................................................ 129 Table 6.92 MANOVA to test the affect of group workload scales on main effects and interactions....... 129 Table 6.93 Variance analysis for the value of group interaction .............................................................. 130 Table 6.94 Multiple comparisons of value of group interaction in each design phase ............................. 130 Table 6.95 Variance analysis for the difficulty of group interaction ........................................................ 130 Table 6.96 Multiple comparisons of difficulty of group interaction in each design phase....................... 130 Table 6.97 Variance analysis for degree of cooperation........................................................................... 131 Table 6.98 Multiple comparisons of degree of cooperation in each design phase.................................... 131 Table 6.99 Variance analysis for overall group workload ........................................................................ 131 Table 6.100 Multiple comparisons of the mean overall group workload based on phase ........................ 132 Table 6.101 MANOVA to test the affect of reflective group workload scales......................................... 132 Table 6.102 ANOVA for the reflective overall group workload .............................................................. 132 Table 6.103 Multiple comparisons between project support types for overall group workload............... 132 Table 6.104 MANOVA to test the group workload scales assessed by outside observers during planning .......................................................................................................................................................... 133 Table 6.105 MANOVA to test the group workload assessed by outside observers on the main effects and interactions during design ................................................................................................................ 133 Table 6.106 ANOVA for difficulty of group interaction evaluated by outside observers ........................ 133 Table 6.107 Comparisons of difficulty of group interaction evaluated by external observers ................. 133 Table 6.108 ANOVA for overall group workload evaluated by outside observers .................................. 134 Table 6.109 Comparisons of overall group workload evaluated by external observers ........................... 134 Table 6.110 MANOVA for reflective group workload scales assessed by outside observers.................. 134 Table 6.111 Percentage of effective and ineffective behaviors summarized by design phase ................. 135 Table 6.112 Percentage of effective and ineffective behaviors summarized by project support level ..... 136 Table 6.113 MANOVA to test the affect of all effective and ineffective behaviors during planning ...... 137 Table 6.114 ANOVA for all ineffective behaviors during planning......................................................... 137 Table 6.115 ANOVA for all effective behaviors observed during planning ............................................ 137 Table 6.116 MANOVA to test the affect of the critical team behaviors on main effects and interactions during planning ................................................................................................................................ 137 Table 6.117 ANOVA for effective coordination behaviors during planning ........................................... 138 Table 6.118 ANOVA for effective giving feedback behaviors observed during planning....................... 138 Table 6.119 MANOVA to test the affect of all effective and ineffective behaviors on main effects and interactions ....................................................................................................................................... 139 Table 6.120 Variance analysis for all ineffective behaviors ..................................................................... 139 Table 6.121 Multiple comparisons of ineffective behaviors in each design phase................................... 139 Table 6.122 Multiple comparisons of ineffective behaviors based on role .............................................. 139 Table 6.123 Variance analysis of all effective team behaviors................................................................. 140 Table 6.124 Multiple comparisons of effective behaviors in each design phase ...................................... 140 Table 6.125 Multiple comparisons of effective behaviors based on role.................................................. 140 Table 6.126 MANOVA to test the affect of critical team behaviors on main effects and interactions .... 141 Table 6.127 Variance analysis for ineffective communication behaviors observed during design .......... 141 Table 6.128 Comparisons between design phase and project support for ineffective communication .... 142 Table 6.129 Variance analysis for effective cooperation behaviors ......................................................... 143 xii Table 6.130 Multiple comparisons of effective cooperation in each design phase................................... 143 Table 6.131 Multiple comparisons of effective cooperation based on role .............................................. 143 Table 6.132 Variance analysis for effective coordination behaviors observed......................................... 144 Table 6.133 Multiple comparisons of effective coordination in each design phase ................................. 144 Table 6.134 Variance analysis for effective giving feedback behaviors observed ................................... 144 Table 6.135 Multiple comparisons of effective giving feedback in each phase ....................................... 145 Table 6.136 Variance analysis for effective adaptability behaviors ......................................................... 145 Table 6.137 Multiple comparisons of effective adaptability behaviors in each design phase .................. 146 Table 6.138 Multiple comparisons of effective adaptability behaviors based on role.............................. 146 Table 6.139 Comparisons of effective adaptability behaviors for project support levels......................... 146 Table 6.140 Correlations between various critical team behaviors and dependent performance variables .......................................................................................................................................................... 146 Table 6.141 Correlations between various critical team behaviors and the NASA TLX during planning147 Table 6.142 Correlations between various critical team behaviors and job satisfaction during planning 147 Table 6.143 Correlations between group workload and critical team behaviors during planning............ 148 Table 6.144 Correlations between critical team behaviors and the NASA TLX during design ............... 148 Table 6.145 Correlations between various critical team behaviors and job satisfaction .......................... 149 Table 6.146 Correlations between various critical team behaviors and group workload scales............... 149 Table 6.147 Correlations between various critical team behaviors and design performance ................... 150 Table 6.148 MANOVA to test the affect of supplemental group observations on main effects and interactions during planning............................................................................................................. 150 Table 6.149 ANOVA for time-related comments during planning .......................................................... 150 Table 6.150 ANOVA for the money-related comments during planning................................................. 151 Table 6.151 Multiple comparisons between roles for money-related comments ..................................... 151 Table 6.152 ANOVA for the non-task related comments during planning .............................................. 151 Table 6.153 MANOVA for supplemental group observations during design .......................................... 151 Table 6.154 Variance analysis for time-related comments....................................................................... 152 Table 6.155 Multiple comparisons of time-related comments in each design phase................................ 152 Table 6.156 Multiple comparisons of time-related comments based on role ........................................... 152 Table 6.157 Multiple comparisons of time-related comments based on project support and role............ 153 Table 6.158 Variance analysis for money-related comments ................................................................... 154 Table 6.159 Multiple comparisons of money-related comments in each phase ....................................... 154 Table 6.160 Multiple comparisons of money-related comments based on role........................................ 154 Table 6.161 Comparisons of money-related comments based on project support and phase................... 154 Table 6.162 Variance analysis for non-task-related comments ................................................................ 155 Table 6.163 Multiple comparisons of non-task-related comments in each design phase ......................... 155 Table 6.164 Comparisons of non-task related comments by phase and project support .......................... 156 Table 6.165 Correlation between doubt and various measures of design performance............................ 157 Table 6.166 Correlation between time-comments, perceptions of time and temporal demand ................ 157 Table 6.167 Correlation between difficulty of group interaction and perceptions of competence and helpfulness........................................................................................................................................ 157 Table 6.168 Correlation between average TLX and overall group workload........................................... 157 xiii List of Figures Figure 1.1 Overview of the experimental design .......................................................................................... 4 Figure 2.1 A life-cycle model for engineering analysis and design............................................................ 10 Figure 3.1 Relationship between number of replicates and the standard minimum difference (*) .......... 36 Figure 3.2 Diagram of the system test site.................................................................................................. 39 Figure 4.1 Box and whisker plot of age range for treatment groups........................................................... 49 Figure 4.2 Grade point average range for each treatment........................................................................... 50 Figure 4.3 Academic level of the participants ............................................................................................ 51 Figure 4.4 Box and whisker plot of self-reported participation in engineering design projects ................. 51 Figure 4.5 Box and whisker plot of self reported familiarity with project management (PM)................... 52 Figure 4.6 Average time spent in each design phase based on team design ............................................... 58 Figure 4.7 The mean TLX ratings for the interaction between design phase and project support.............. 62 Figure 5.1 Comparison of mental demand for the interaction between team design and design phase...... 72 Figure 5.2 Comparing effort ratings for the interaction between design phase and project support .......... 75 Figure 5.3 Interaction between project support and team design for physical surroundings...................... 79 Figure 5.4 Comparisons of developed ability perceptions for team design and project support ................ 82 Figure 5.5 Comparing reflective developing ability for team design and project support.......................... 86 Figure 6.1 Comparing mean TLX ratings for the interaction between design phase and project support 100 Figure 6.2 Comparing performance ratings for the interaction between design phase and project support .......................................................................................................................................................... 103 Figure 6.3 Comparisons of challenge for the interaction between project support and role..................... 107 Figure 6.4 Comparison of level of interest in the problem based on project support and role ................. 114 Figure 6.5 Comparisons of resources for the interaction between project support and role..................... 116 Figure 6.6 Comparison of group member competence based on project support and role....................... 118 Figure 6.7 Comparisons of helpfulness for the interaction between project support and role.................. 119 Figure 6.8 Comparison of reflective interest for project support and role................................................ 121 Figure 6.9 Comparison of reflective problem difficulty for project support and role............................... 122 Figure 6.10 Comparison of ineffective communication for design phase and project support................. 142 Figure 6.11 Comparisons of time-related comments for the interaction between project support and role .......................................................................................................................................................... 153 Figure 6.12 Comparing the money-related comments for project support and design phase ................... 155 Figure 6.13 Comparing non-task related comments for project support and design phase ...................... 156 xiv Chapter 1 Introduction Based on customer demand, industry is rapidly introducing products into the market at lower costs (Frankenberger & Auer, 1997; Ragusa & Bochenek, 1996). To meet the customers expectations, organizations have turned to teams to take advantage of the expertise in the current workforce. A team has several individuals that work on tasks which are components of a larger project. The individuals each contribute a diverse set of knowledge and skills and frequently have extreme pressure to finish within specific time guidelines. This definition of team was provided by Orasanu and Salas (1993) and is very similar to that of a project team, which is a special case due to the time period within which the members that are part of the team (Pinto & Pinto, 1991). While there is evidence that teams are effective, due to the social complexity of teams, effectiveness has not consistently translated across teams (Salas & Cannon-Bowers, 2000). Because teams consist of individuals with various backgrounds and personalities, each team is slightly different. An advantage of teamwork is that the presence of other individuals may increase arousal, which has been found to have a positive effect on performance (Medsker & Campion, 1997; Zajonc, 1965). The effect of arousal might offset boredom and decrements in performance. Prior studies have found that group work can produce negative effects as well. For example, many researchers believe that teamwork adds a level of mental workload beyond the workload due to the task work (Bowers, Braun, & Morgan, 1997). If one member of the team is overloaded, the performance of the entire team tends to degrade (Roby & Lanzetta, 1957). From research conducted in the 1960s, the increased communication requirements due to teamwork were found to increase mental workload (Johnston, 1966; Naylor & Briggs, 1966; Williges, Johnston, & Briggs, 1966). This line of research invited skepticism on the usefulness of teamwork. Through his research, Kidd (1961) found that performance did not improve as group size increased when comparing individuals, dyads, and triads. However, Kidd acknowledged group efforts might be beneficial for long-term projects when considering the intrinsic motivation generated by individuals working in groups. Supporters of collaborative design projects argued the benefits include an increased knowledge base and potential cost savings, which reduce the potential for error during implementation (Kidd, 1994). Design teams are being used to help reduce cycle times for the introduction of products to market. The majority of the studies conducted on teams working on engineering design projects have focused on a narrow aspect of design or project management, for example, the use of design support or communication support and how teams with various structures perform during planning. The tasks involved with projects are twofold: one component is the actual deliverable (or project) and the other component is the management of the personnel, resources, and tasks involved with the project. Recently, software tools have been introduced with the specific goal of managing projects. While there survey studies have compared the use of various project management software tools in industry, there is no evidence that automating project management tools improves the projects being managed. Prior work has been conducted indicating which project management tools do improve project performance, however this research considered paper and pencil (manual) approaches and did not compared manual with software-based (automated) approaches (Liberatore & Titus, 1983; PollackJohnson & Liberatore, 1998). While the trend in industry has been towards using teams or groups for engineering design projects, little work has been conducted to justify the use of groups working specifically on engineering design projects. The structure of project teams in the planning phase of project management has been studied under controlled conditions (Kernaghan & Cooke, 1986; 1990). Individuals and various types of groups were compared to determine, which type of structure could arrange project management activities in a logical order. However these studies are limited in nature and participants did not actually plan or implement an actual project. 1 The need to focus on management of projects was highlighted in a recent field study looking at a project team with members located Europe, Mexico, and the United States (Kayworth & Leidner, 2000). One of the main problems that emerged from this study was the teams difficulty with managing the project. 1.1 Problem Statement To compete and survive, organizations are considering ways to improve the design cycle time. Designs are for either the creation of a product or a process. Inherent in a design project is the investigation of the initial stages of the engineering design life-cycle which overlaps with several phases of the project life-cycle. The creation of an engineering design lends itself to being managed as a project. How performance compares between design projects that are planned followed by continuous monitoring and those that are not managed need investigation. Although a common characteristic in engineering design and project management research is the use of teams, little guidance exists on how and when engineering designers should collaborate. The combined effect of collaboration and project support is an unresolved issue. 1.2 Research Objectives The research objective for this dissertation was to study the effects of altering team design between individuals and groups of three and the type of project support, no project support, manual project support or automated project support. The effects were measured by collecting subjective mental workload and job satisfaction data. In addition, traditional performance measures, cost effectiveness and design cycle time were assessed. The practical goal of this research was to improve the engineering design life-cycle through understanding the effects of providing project management support and guidelines for people interacting throughout the design project life-cycle. While the current trend is to use groups to manage engineering design projects, little research had been conducted to verify that groups are the most effective throughout the design projects life-cycle. Furthermore, from prior studies, there was evidence that group work added to the level of mental workload beyond the workload from the task alone. The expected outcome of this experiment was a set of guidelines for managing engineering design projects. Practical guidance on how to manage engineering projects more efficiently through manipulating the team design and project management support at specific phases of the design projects life-cycle was the final goal. By comparing performance in engineering designs with and without project support, further justification was expected for the use of project support for engineering design projects. 1.3 Questions and Hypotheses The overriding question for this research was: How does varying team design and project support affect engineering design performance, project performance and the people involved in the engineering design project? This question was partitioned into three sub-questions: How was performance (cost effectiveness and design cycle time) affected by team design and project support during the projects life-cycle? What were the changes in mental workload and job satisfaction due to variations in team design and project support during the projects life-cycle? 2 How was project performance (planning time, cost variance and schedule variance) affected by team design and manual versus automated project support? Based on a review of the literature, several hypotheses were generated, which were formally tested: Hypothesis 1: Groups would take longer to complete the design project than individuals. Treatments with manual project support would take longer to complete compared to the other treatments. Treatments with project support would have more cost effective designs than those in without project support. Groups would have more cost effective designs than individuals. Mental workload would be lower for groups than individuals. Mental workload would be higher in treatments in which participants use automated project support compared to treatments where participants used manual support or no support. Job satisfaction would be higher for groups than individuals. Job satisfaction would be similar in all project support treatments. The planning time would be longer for groups compared to individuals. Planning time would be longer for treatments with manual support compared to automated support. Cost variance and schedule variance, would be inferior for treatments with manual project support compared to those with automated support. Cost variance and schedule variance would be inferior for individuals compared to groups. Hypothesis 2: Hypothesis 3: Hypothesis 4: Hypothesis 5: 1.4 Research Overview Sociotechnical systems (STS) theory provided the framework for this research. From STS, a systematic approach to identifying and organizing factors of interest was developed. Sociotechnical systems underscored the importance of applying a systems perspective to this problem ensuring that all aspects of the system were considered. Figure 1.1 contains the overview for this study. The consensus from STS theory was that an organizational system consists of the personnel subsystem, the technological subsystem, the organizational design, and the environment in which the organization exists (Pasmore, Francis, Haldeman, & Shani, 1982). The personnel subsystem consists of the people and relationships between the people within the organization. In this study, the personnel subsystem consisted of engineering designers who were working on engineering design tasks as shown in Figure 1.1. The technological subsystem refers to the method for producing work and the tools and techniques involved. For this study, the factor of interest was the type of project support: no support (the control), manual tools for planning and tracking the project, and automated tools for planning and tracking the project. The organizational design refers to how the work system is structured and the coordination and communication processes that are in place within the work system. The team design was the factor of interest in this research. Individuals working independently and groups of three working together in the same physical location are the team designs that were compared. The process was investigated in this research was part of the engineering design life-cycle, which included conceptual design, preliminary design, and detailed design. Groups and individuals with project support also participated in a planning phase, which occurred prior to the conceptual design. This was operationalized with groups and individuals using project management tools to plan their design process. Executing the plan is analogous to the implementation stage in the engineering design life-cycle. When participants did not have project support they were not instructed to plan their activities. While the 3 projects were implemented, participants with project support were required to monitor their progress. Again participants in the control did not track progress on the design project. Project Lifecycle: Project Planning Project Implementation Engineering Design Lifecycle: Conceptual Design Preliminary Design Detailed Design Technological Subsystem Project Management Tools none manual automated Manual Engineering Design Tools Personnel Subsystem Design Project Personnel Technological Variables Primary Task Measures Design Performance (design cycle time, lifecycle cost and system effectiveness) Project Performance: planning time, cost & schedule variance Personnel Subsystem Individual Mental Workload (NASA-TLX) Organizational Structure Job Satisfaction Project Monitoring and Control Team Design Individuals Groups of Three Group Process: Critical Team Behaviors (communication, coordination, cooperation, team spirit and morale, ability to give/receive feedback, and adaptability) Group Workload Environment (uncertainty, available resources) Independent Variables Process Dependent Variables Figure 1.1 Overview of the experimental design (adapted from Kraemer et al., 1990; McGrath et al., 1994; Salas et al., 1992) The dependent variables were selected based on previous research that used a STS framework (for example, Grenville, 1996; Hacker, 1997; Meredith, 1997; Trist & Bamforth, 1951). From these studies, a commonality was the need for a balanced set of measures representing the technological and personnel subsystems and the organizational design. The measures of technological performance included cost effectiveness and design cycle time. Planning time, cost variance, and schedule variance were project performance measures that were collected for treatments that included manual or automated project support. Mental workload was evaluated for the personnel subsystem. The organizational structures dependent variable was job satisfaction. When the team design was groups, participants completed a group workload scale. Videotapes of the interaction between group members were observed for several team behaviors. 4 Chapter 2 2.1 Literature Review The Macroergonomic Perspective Macroergonomics is concerned with designing organizational and work systems taking into account the pertinent sociotechnical variables and the interactions between these variables (Hendrick, 1986). The macroergonomic perspective begins by identifying and analyzing the personnel, technological, and environmental variables and assessing these variables implications on the work systems design. The work system design carries through to the human-job, human-machine, and the human-software interface design (Hendrick et al., 2000). Organizations are bounded systems operating with a purpose or goal and have transformational processes that convert inputs into outputs (DeGreene, 1973; Pasmore et al., 1982). Historically, organizations have been viewed as the planned coordination of two or more people who, functioning on a relatively continuous basis and through a division of labor and a hierarchy of authority, seek to achieve a common goal or set of goals (p. 5). (Robbins, 1983) However, with the introduction of communication technology and shared computer applications, organizations are turning to a virtual definition which means that the people work together for brief periods of time to accomplish a specific task (Cano & Kleiner, 1996). Three work system design practices that have been found in dysfunctional work system design include technology-centered design, leftover approaches to function allocation, and failure to integrate the sociotechnical systems characteristics into the design of the work system (Hendrick, 1997). Macroergonomics, on the other hand, is a human-centered approach to organizational and work system design. The expectation is macroergonomic interventions will show improvements of 60% 90% in employee job satisfaction, accident rates, and quality measures. Support for this claim was reported in previous studies using macroergonomic approaches. Several macroergonomic redesign efforts were achieved in two companies (in separate interventions) in New York State in a partnership between the State of New York, a company, and a university (Kleiner & Drury, 1999). In one company, the intervention resulted in the creation of a new plant. Comparing the organizational results of the new plant to the old plant resulted in a 75% changeover time reduction, $3 million inventory reduction, 95% quality improvement, a cycle time reduced by 50-70%, and increased employee satisfaction. In the second company, the intervention was the creation of a training program. This intervention resulted in improved communications, improved teamwork, improved safety, and a 3-month return on investment. In another set of studies by Nagamachi and Imada (1992), injuries and vehicle accidents were reduced between 76% and 90% percent. Significant cost savings based on the medical costs and worker compensation was reported in participative ergonomics efforts, which are in the macroergonomic research domain (Imada & Bowman, 1992). In a study by Rooney, Morency, and Herrick (1993), lost time due to accidents and injuries was reported to decrease by over 70% (as cited in Hendrick, 1997). Other success has been reported by Kleiner (1996), for example General Mills decreased their grievances by 69% in one year and were able to develop a new organizational culture. 2.1.1 Sociotechnical Systems Theory By definition, the sociotechnical perspective considers organizations as open systems (Pasmore et al., 1982; Hanna, 1988; Hendrick, 1986). According to open system theory, when changes occur in the environment, the effects will be felt by all subsystems. The personnel and technological subsystem respond jointly to causal events, which is the principle of joint causation (Hendrick, 1986). To design one subsystem without considering the other subsystems would result in a sub-optimized system. Therefore, the personnel and technological subsystems must be designed together to result in joint optimization. The personnel and technological subsystems with the external environment will influence the organizational 5 structure. Similarly changes in the organization structure will affect the personnel and technological subsystems. Each of the subsystems will be analyzed to reduce the potential for sub-optimization during the engineering analysis and design process. Sociotechnical design principles were formally published by Cherns in 1976 and have since been modified (Cherns, 1987). These principles are guidelines for engineers involved with designing sociotechnical systems. To summarize these guidelines, design should be completed with minimal task specification, involve the appropriate personnel and provide the personnel with the appropriate tools, resources and information needed to accomplish their task. Sociotechnical systems theory is rooted in the work conducted by Trist and Bamforth (1951) on the longwall method of coal mining in the United Kingdom. Prior to their study, the mining system was a traditional, manual system where work was conducted using small independent groups. The control of the task was mainly left to the group. Each group was responsible for a small section of the coal wall (referred to as the shortwall method of coal getting). In the late 1940s and early 1950s mechanization was introduced to the coal-mining industry, specifically coal-cutters and conveyor belts. This mechanization enabled the miners to work on longer sections of the coalface. With the mechanization also came the need for a new type of work unit, cycle groups consisting of 40-50 men. This change disrupted the harmony that previously existed with the manual labor, autonomous small groups. The result of this new system was an increase in absenteeism, inter-group rivalry, decreased productivity, and lower production. In subsequent studies of coal mining, a composite longwall system was compared with the conventional longwall method (DeGreene, 1973). The composite longwall method reintroduced the concept of autonomous workgroups back into coal minings organizational design. Thus, the composite longwall method increased skill variety, created the opportunity for satisfying social needs, allowed for group selection by workers, and reduced the interdependences between the three shifts. The result of this intervention was a significant increase in productivity, reduced absenteeism, and dissatisfaction compared to that of the conventional longwall method. An important outcome from the Tavistock studies was that work system must be designed such that there is congruence between the people who are involved in the system, the technology that is used to perform the work, and the external environment in which the system operates (Hendrick et al., 2000). Since the Tavistock studies (the set of coal mining studies is often referred to as the Tavistock studies since Trist and Bamforth were associated with the Tavistock Institute of Human Relations at the time of the original studies), evidence continues to grow in support of sociotechnical systems research. Some of the reported outcomes of sociotechnical interventions include decreased costs, improved quality and attitudes, increased productivity, reduced turnover, absenteeism, and injuries (Pasmore, 1988). In a review of 134 sociotechnical systems experiments, Pasmore, Francis, Haldeman, and Shani (1982) found the vast majority of the experiments to be successful. By far the most used feature of sociotechnical systems theory from this research was autonomous work groups (53%) followed by technological skill development (40%). Productivity increases due to sociotechnical systems research have been found in blue-collar industrial settings (Pasmore et al., 1982; Cummings, Molloy, & Glen, 1977), and in whitecollar environments (Beekun, 1989; Cummings et al., 1977). The relative amounts of increased productivity were similar between the blue- and white-collar environments, based on a meta-analysis of seventeen sociotechnical systems studies (Beekun, 1989). In an analysis of 58 sociotechnical experiments, threats to external and internal validity of performance and attitudinal findings were explored (Cummings et al., 1977). One potential limitation of the meta-analyses is the lack of reported failures of sociotechnical research. However, the sociotechnical improvement methods were generalizable to a variety of organizational settings, regardless of gender, hierarchical level, unionization, or size. Internal validity was a general concern for the studies included in the analysis. For both performance and attitudinal findings, the biggest threat to internal validity was mortality (in other words, subject withdrawal). 6 2.1.2 Function Allocation Function allocation is the assignment of work between humans and machines (Price, 1985; Kantowitz & Sorkin, 1987), and is often a consideration in macroergonomic studies. The earliest literature on function allocation comes from Fitts (1951). Fitts list was an attempt to systematically approach the assignment problem by providing a characterization of functions that are better suited to be performed by a machine and functions that a human is uniquely qualified to complete. In developing this list, Fitts assumed that each function could be completed by a human or by a machine, but not a combination of the two. With computer hardware and software that allows humans and machines to share functions, the function allocation problem can be very complex (Rouse & Cody, 1986). In addition to Fitts list, there are several common assignment strategies that designers use for function allocation (Welch, 1994). One strategy is to assign tasks to the human or machine based on economic considerations. Another strategy is to assign every possible function to a given element, for example the machine, and then assign the left over functions to the other element, for example, the human. These two methods of assignment leave much to be desired. In systems designed with a macroergonomic perspective, the goal is to have functions (or tasks) that are appropriately allotted to the machine and human such that the personnel subsystem and the technological subsystem complement one another. Tasks should not be automated for the sake of automation. Trade offs must be considered in order to achieve a balance between performance resources (humans and technology) and performance demands, for example safety and efficiency (Hollnagel & Bye, 2000). Function allocation will be revisited after the personnel and technological subsystems are discussed. 2.2 Personnel Subsystem The personnel subsystem describes the ways in which the task are performed (Hendrick, 1986). People within the organization and their relationships are included in the personnel subsystem (Pasmore et al., 1982). The personnel within an organization should be analyzed according to their degree of professionalism and psychosocial factors (Hendrick, 1986). Analyzing the personnel based on their degree of professionalism (or education and training of the workforce) provides insight for the organizational and work system design (Hendrick, 1986). Through education and training, engineers tend to have a high degree of professionalism. An engineers professionalism is an internalized formalization of behavior; values, norms, and expected behaviors learned prior to joining an organization, even though this behavior is reinforced after entering the workforce (Hendrick, 1986). Psychosocial factors describe the degree to which a culture or subculture provides an opportunity for exposure to diversity thorough affluence, education, communication, and/or transportation. The more a person is actively exposed to diversity, the more cognitively complex the person will become (Hendrick, 1986). Cognitive complexity refers to the underling different conceptual systems for perceiving reality (p. 474). (Hendrick, 1986) When individuals have low cognitive complexity, they require structure, stability, and order in their environment. However, as cognitive functioning becomes more abstract, there is less need for structure. Hendrick (1997) found evidence that concrete workgroups and managers functioned well under high centralization, vertical differentiation, and formalization; while cognitively complex work groups and managers functioned well with low centralization, low vertical differentiation, and little formalization. Another important aspect of the personnel system is the individuals response to workload. 7 2.2.1 Workload Workload consists of both mental and physical components. Physical work occurs when the body expends energy through muscular contractions (Sanders & McCormick, 1993; Rodahl, 1989). The demands that are met through physical work include moving parts of the body, moving objects that are not a part of the body, and maintaining posture (Kilbom, 1990). Historically, researchers have not agreed on a definition of mental workload (Gopher et al., 1986; Sheridan, 1979). According to Moray (1988), validity is a prerequisite for a sound theory, which requires the existence of precise operational definitionsno such definitions exist (p. 125). A framework for defining the definitions of workload, both mental and physical, and the relationships of the definitions to the various measures of workload was developed by Sheridan (1979). He identified six alternative definitions of workload occurrences: assigned task; performance measures; information an individual processes to complete a task; energy an individual expends to complete the task; the individuals emotional state; and individuals performance. Sheridan qualified several of these definitions. Sheridan preferred information processing requirements and the emotional state as the major components of workload. Four measures of workload were defined based on the six workload definitions. Subjective judgments (typically operationalized as ratings) and physiological indices are based on the information processing, energy expended and the emotional load definitions of workload. The assumption underlying subjective judgments is that an individual can accurately rate his or her workload level (Svensson, Angelborg-Thanderz, & Sjober, 1993; Borg, 1971). For physiological measures, the assumption is a relationship exists between the physiological reaction and task demands (or workload level) (Svensson et al., 1993). If the assigned task includes a secondary task with an associated performance criterion, the secondary task performance indicates the level of work associated with the primary task. Primary task measures consider the objective performance based on a single criterion or set of criteria. Decreases in the performance level are assumed to be the result of increases in mental workload (Svensson et al., 1993). More recent literature on mental workload tends to agree with Sheridans preferred definition, that workload is related to information processing capabilities (Wickens, 1979; Gopher et al., 1986; ODonnell et al., 1986; Eggemeier, 1988; Grise, 1997). Processing capacity available to perform a task is considered to be a resource and the processing is the demand (Gaillard, 1993). Similar to this point of view, Sanders and McCormick (1993) purported that mental workload is based on the relationship between the number of resources a person has available and the number of resources required by a specific circumstance. This approach to mental workload helps identify how mental workload fits into other complex theories like stress. Workload from this frame of reference becomes a source of stress because resources are being consumed through the completion of the task. 2.3 Technological Subsystem The technological subsystem consists of the task to be performed and the tools and methods required for the humans to perform the task (Hendrick, 1986). In this research, a management process, the project management life-cycle, and a technical process, the engineering analysis and design life-cycle, were of interest. Two dimensions upon which a task can vary include task variability and task analyzability (Perrow, 1967). Task variability refers to the number of exceptions encountered while completing a task. Task analyzability refers to the type of search procedures available to respond to the exceptions. Four categories of task type result from these dimensions: routine (routine and analyzable), craft (routine and unanalyzable), exceptions (high variety and analyzable), and non-routine (high variety and unanalyzable). 8 2.3.1 Engineering Analysis and Design Engineering design problems tend to be defined as open-ended and ill-structured problems. Because logical solutions can be found using rational processes, engineering design tends to fall into the exceptions category according to Perrows classification (Simon, 1973; Malhotra, Thomas, Carroll, & Miller, 1980; Ball, Evans, & Dennis, 1994; Dym, 1994). However, in some industries, for example aerospace, many of the design applications are non-routine (Hendrick et al., 2000). Engineering design is separated from craft in that crafting is associated with constructing and creating an output, where as engineers and designers create the representation of an output for someone else to construct (Dym, 1994). The design process has been defined as including the following activities: identification, definition, search, criteria establishment, alternative consideration, analysis, decision, specification, and communication (Kurstedt, 1993). Many definitions of engineering design exist in the literature (Dym, 1994; Pahl & Beitz, 1996). Generally, the literature agrees that engineering design is a goal directed decision-making activity that is iterative (Asimow, 1962; Dym, 1994). Typically engineering design involves the creation of a model or pattern that will serve as a template to be replicated as needed, to satisfy a human need (Asimow, 1962). Engineering design is open-ended; therefore the goal can be achieved through many routes (Ferguson, 1992). Engineering design methodologies have been widely addressed by academicians and practitioners alike. Reports that at least 70% of product costs are defined during the early stages of design contributes to the interest in engineering design methods (Evans, 1990; Bertodo, 1994; Ragusa et al., 1996). From the macroergonomics literature, there is a need for design to include ergonomics at all points throughout the life-cycle (Dockery & Neuman, 1994). Two approaches can resolve this issue: one that is very expensive and time consuming is to train designers in manufacturing, handling, transportation, service and other relevant functions. The other approach is to develop a design process involving representatives from all aspects of the life-cycle as part of the design team. By using a participative approach to engineering design, there is accountability, for example functional objectives become apparent early in the design process. Numerous engineering analysis and design methodologies have been developed (Blanchard & Fabrycky, 1990; Pahl et al., 1996; Dym, 1994; Dieter, 1991). The engineering analysis and design method used in this research comes from Blanchard and Fabrycky (1990). This model promotes a system life-cycle approach to design, from conception to phase out. The phases of the design process considered in this research are shown in Figure 2.1. Blanchard and Fabryckys model is a six-phase model and includes conceptual design, preliminary design, detail design, production and construction (manufacturing), utilization and support, and phase-out and disposal. The six phases ensure designers plan and develop the product or system considering all aspects of the life-cycle including maintenance and support, engineering functions, and disposal. The design process is initiated with a stated need and the first phase is to create a conceptual design. During the conceptual design phase, performance parameters and operational requirements are established and conceptual ideas are collected. Previous research indicates designers had better designs when they spent more time on the goal analysis and task clarification compared to the other tasks in the design process (Ehrlenspiel & Dylla, 1993). The purpose of the second phase, preliminary design, is to develop feasible alternative initial concepts that will satisfy the criteria specified during the conceptual design phase. Research supports that higher performing designers developed more alternatives compared to the lower performing designers (Ehrlenspiel et al., 1993). At the end of the preliminary design phase a single concept is selected for detailed design. In the third phase, detailed design, engineering plans, drawings, and other formal documentation are developed such that the product can be created. Phase 4, production and construction, begins after the design has been fixed. 9 Preliminary Design Idea reduction System and subsystem trade offs and evaluation of alternatives Concept selection Preliminary Design (synthesis): layout, parts, cost Prototypes Design Review Detail Design Design drawings Design documentation Instructions Bill of materials Design Review Conceptual Design Identify need Determine design criteria Idea generation Design review Figure 2.1 A life-cycle model for engineering analysis and design (adapted from Blanchard et al., 1990) The engineering functions in the production phase (phase four) include facility design, manufacturing process design, materials selection and inventory requirements, special tools, equipment, transportation and handing equipment design, work methods design, and operations evaluation. Phase five, utilization and support functions, includes system use throughout the intended life-cycle, modifications for improvement, and logistic support for deployment. The engineering functions include involvement with initial deployment and installation, servicing the system for regular operation and maintenance support, installing modifications, and assessment of system operations by collecting and analyzing data. The final phase is the phase out and disposal of the product. The engineering functions are to ensure correct material disposal procedures are employed to minimize the effect on the environment. Phases four, five, and six were not formally included in this research as denoted with dashed lines in Figure 2.1. Much research conducted on engineering design consists of observational studies; for example, researchers used protocol analysis to structure the output of observational studies (Ullman, Stauffer, & Dietterich, 1987; Ullman et al., 1990; Malhotra et al. 1980; Frankenberger et al., 1997; Tang, 1991; Goel & Pirolli, 1989; Schon, 1992). The nature of these studies has been to document the process used in design and explore the use of drawing in design. Subjects participating in these studies have ranged from relatively inexperienced undergraduates to experienced designers in industry. From observational studies, researchers have gained insight into the engineering design process. In a study by Ullman et al. (1987) that focused on the use of drawing in the design process, six uses were found for drawing (adapted from p. 64): recording geometry; communicate ideas among designers and between designers and manufacturers; simulate ideas; for analysis/calculations; verify completeness; and to support the designers memory. They found that often designers would re-design components they had previously designed, which was thought to be caused by a cognitive overload. From a more recent study by Ullman et al. (1990), drawing was found to be a method used to archive and communicate the completed design to manufacturing personnel. The authors emphasized the importance of informal sketches that designers used in the early part of design to assist in creating and recording ideas in addition to communicating ideas between designers. Ullman adapted Newell and Simons (1972) model of the design environment to represent the way designers used drawing to assist their memories. Design occurred both in the external environment (outside the designers mind) and in the internal environment (inside the designers mind). Although this model has not been completely accepted in the cognitive psychology community, it does adequately represent some of the functions of drawing and sketching in the design process. One of the findings from Ullman et al. (1990) with implications for developing a workstation for engineering design was related to why designers used sketches. They observed that sketches tended to 10 occur rapidly. Time lost from a complicated drawing method might result in the loss of an idea. Because sketches were quick and flexible, Ullman believed the sketches were a method for extending memory. Therefore, workstations for collocated and distributed design should incorporate a method for designers to share quick sketches. More recently, there is interest in determining how to support virtual design teams, which is due to innovations in database and communication technology, availability of groupware systems, shared applications, and a drop in computer costs (Reddy et al., 1993). The computer environment should support spontaneous and planned communication, coordination activities, in addition to design activities. There are many efforts underway to make these types of systems a reality. 2.3.2 Project Management Projects have a purpose, a specific time frame and budget, use resources, and require a series of interrelated tasks to accomplish the purpose (Knutson & Bitz, 1991; Weiss & Wysocki, 1992; Gido et al., 1999). Projects have definable deliverables that can be unique or routine (Moder, Phillips, & Davis, 1983). Inherent to projects is a degree of uncertainty. Uncertainty arises from a need to estimate time, resources, and costs required to complete a project and from the capability of the resources. Project management is a set of tools, methods, and techniques used to propose, plan, implement, control, and terminate the progress of goal-oriented work within a specific organizational context (Knutson et al., 1991). There are a variety of project management processes that have been reported in the literature. The project management cycle that was used for this research comes from Gido and Clements (1999). The four phases of the project management process include: Phase 1, identification of the need, problem, or opportunity; Phase 2, development of a solution (a plan of action); Phase 3, implementation (performing the project); and Phase 4, project termination. The functions that are associated with project management include defining, planning, organizing, and controlling. This is a process with a definite start and end (Kliem & Ludin, 1998). The first phase is to determine the overall goal and objectives for the project. During this phase, the scope is defined, resources are identified, and a short document is developed that stakeholders sign agreeing to general definition of the project. The scoping document contains a statement of the problem or opportunity, the overall goal, the specific objectives, success criteria and any assumptions that the project team makes regarding the project (Weiss et al., 1992). The second phase is to plan the project. During this phase, schedules are developed, resources are assigned, and costs are estimated. There are a number of tools that can be used to facilitate planning. Examples include the use of a Gantt chart (a bar chart that portrays the plan, schedule, and progress of a project) and a network diagram or PERT chart (displays relationships between tasks), a work breakdown structure (a hierarchical layout of all tasks and deliverables associated with the project), a responsibility matrix (shows individual responsibilities for each task), and milestone logs (indicates important activities and dates). The third phase is putting the second phase into action. During this phase, there are status reports and changes that occur to the baseline schedule and budget are managed. Effective communication and coordination were reported to be keys to this phase (Gido et al. 1999). The final phase is ending the project. At the end of the project, there is delivery of the outcome, assessment of the baseline compared to the actual time frame and cost, resources are released and the project is evaluated to determine what was learned. The focus of this research was on Phase 1, scoping, Phase 2, planning, and Phase 3, implementation (tracking and controlling the project). The use of manual project support tools, for example, hand drawn Gantt Charts, the critical path method, and other scheduling and budgeting tools have been well established and supported in the literature (Moder et al., 1983; Archibald, 1976). Many tools have recently been automated. Examples of automated tools include software packages that assist in planning, calculating the critical path and report development to desktop videoconferencing for team status report meetings. Survey research has been 11 conducted to determine the percent of organizations that use project management and the most commonly used commercial software packages. From a study conducted by Pollack-Johnson and Liberatore (1998), 90% of the survey respondents primarily used project software for planning and 80% used software for project control. About 8% of the respondents did not use software for project management activities. In Pollack-Johnsons study, the most frequently used project management software package was Microsoft Project (50% of respondents). In a different survey, Microsoft Project was again found to be the most frequently used computerized tool (48% of respondents, next closest was Primavera Project Planner reported by 13.8% of the respondents; Fox & Spence, 1998). A comprehensive literature search did not identify research comparing automated project management tools to manual tools. 2.4 Function Allocation Revisited As mentioned earlier, function allocation is a method for assigning tasks between humans and machines (Price, 1985). In function allocation, systems should be designed to support and not overly tax the human. Performance measures are necessary, but not sufficient, for understanding the system (Wastell, 1991; Wickens, 1984). Hence, a taxonomy of function allocation should provide a basis for classifying the contributions of humans and machines to the overall system. The taxonomy shown in Table 2.1 provides levels of automation with mental and physical workload requirements (adapted from Kleiner, 1998; Kantowitz et al., 1987; Rohmert, 1977). Table 2.1 Taxonomy for the classification of automation levels and operator roles augmented with levels of workload (adapted from Kleiner, 1998; Kantowitz et al., 1987; Rohmert, 1977) Human Dominant 1 Human supplies power, decision making and control 2 Mechanical support for power or control; human decision making Coordinating sensormotor functions 3 Partner 4 Machine supplies power, information, decisions and control; human monitors and/or supplies information Converting information into knowledge Technology Dominant 5 Machine supplies power, information, decisions and control; no monitoring required Producing information Producing force Converting information into action Equal Mental and Physical Workload Physical Workload Dominant Mental Workload Dominant The levels of automation were developed with a focus on the contribution of the human and machine to the overall system (Kleiner, 1998). The lowest level represents the humans responsibility for all aspects of the task, including the decision-making, power, and control. Although this is a low level of automation, mental workload still exists in the system. The next level of automation has the computer supplying the power and/or control, but the human retains decision-making control. At the third level, the computer provides both the power and controls and the human makes the decisions. In the fourth level, the human monitors or supplies information to the computer and the machine supplies power, information, and control. The fifth level is a machine dominant level, where the machine supplies power, information, control, and makes decisions. Workload is an important augmentation of this taxonomy because understanding the humans workload can help designers create better systems (Klein & Cassidy, 1972). As the automation level increases, the workload tends to shift from physical to mental. However, at all levels there is some amount of both mental and physical workload. One of the expectations with automation is that mental demands will be reduced and therefore resulting in fatigue reductions (Harris, Hancock, Arthur, & Caird, 1995). However, this expectation has not always been realized. A potential danger associated with automation arises from the human-computer 12 interface. The interfaces tend to add to the mental load due to the complex sequence of tasks that the human must execute to communicate with the machine. Therefore, to understand how the human performs within the system, the humans mental workload must be known (Price, 1985). In previous research, computer usability issues have been found to negatively affect the human (Baeker et al., 1988). Some of the problems included increased learning times, greater subjective fatigue and stress, increased number of errors committed, and a reduction in time-sharing abilities. One of the explanations for the negative effects of computer usage might be due to individuals need to develop complex mental models in order to use the system. Simply interfacing with a computer might impact working memory (Norman, 1986). Furthermore, computers make demands on attentional processes (Hockey, Briner, Tattersall, & Wiethoff, 1989). Some of the sources of demand include the use of multiple windows, displays of large amounts of data, and communication pipelines through the computer in addition to task work (Hockey et al., 1989; Salvendy, 1981). Computer response delays have been linked with increases in mental strain and stress (Johansson et al., 1984). In studies of computer work, routine users who spent more than 50% of their day entering data reported higher levels of fatigue after work compare to individual who spent approximately 10% of their day on computer tasks (Johansson, 1984; Johansson & Aronsson, 1984). After work, routine users also reported that unwinding after the day took longer than light users. A series of studies were conducted to determine if typing speeds affected a physiological measure of workload (Kohlisch & Schaefer, 1996). In the first experiment, the physical effects of typing were studied by using keystrokes to keep a cursor within a box on the computer screen. Box size and keystroke were manipulated. The results indicated that motor load and mental load were independent. The motor load did not affect the heart rate until keystrokes were less than 150 ms apart. In the follow-up experiment, to study the effect of mental load, a secondary task was assigned in which subjects memorized a sequence of numbers or calculated the sum of the numbers in a sequence. Heart period indicated that there was a main effect for mental and motor load. Heart period was significantly shorter in the addition task than in the memorization task. Heart rate was only affected at typing speeds faster than 360 ms. Considering the average individual has been observed to type at 891 ms 5 s, heart rate periods should not be affected by typing (Kohlisch et al., 1996). However, for individuals who are specially trained in data entry, motor load might be a consideration. Further insight on automation came from a study on the effect of age and experience on mental workload during computer tasks (Nair, Hoag, Leonard, & Czaja, 1997; Sharit, Czaja, Nair, Hoag, Leonard, & Dilsen, 1998). This study took place over three days. Over time, subjective measures of stress decreased, and correlations between frustration, stress, and arousal declined over time. These results generally supported the need for training on computer tasks. Several workload studies have targeted engineering design tools. In a field experiment on computer aided design (CAD) systems, the expectation was that CAD tools would assist the designers, thus reducing physiological and psychological costs and enhancing performance (Luczak et al., 1991). The use of several functions on CAD systems resulted in high perceptions of stress compared to using pencil, paper, and a drawing board. Specifically, the use of the dimensioning function resulted in the greatest stress. In 60% of the trials, using CAD led to higher perceived strain than using pencil and paper. In both studies, tasks were allocated to the computer because the belief was that the humans job would become easier. However, CAD drawings required more attention than pencil and paper which may have added to the designers perceptions of strain (Beitz, Langner, Luczak, Muller, & Springer, 1990). How automating the planning and tracking processes affect participants is not in the literature. Pencil and paper design tools were selected for this study to avoid adding to the workload from the use of CAD. 13 2.5 External Environment Organizations must constantly monitor their environment and develop feedback mechanisms that can sense changes in the environment (Taylor et al., 1993). Organizations need to develop the capabilities to respond to environmental changes (Hendrick, 1986). In the analysis of the environment, environmental uncertainty has been found to impact effectiveness (Burns & Stalker, 1961). Environments are characterized by the degree of change and the degree of complexity (Hendrick, 1986). The degree of change refers to the extent that a given task environment is stable or dynamic over time. The degree of complexity is defined by number of components within an organizations environment. When environments can be characterized as stable and simple, there is little uncertainty in that environment. On the other hand when environments are stable and complex, there is moderately low uncertainty. When the environments become dynamic and simple, there is moderately high uncertainty. And when the environment is characterized by complex with dynamic change, the environment there is high uncertainty. Engineering design environments tend to be unstable and need to maintain flexibility and adaptability. However applying project management to the design process is an attempt at stabilizing the environment. Unstable environments prescribe an organizational structure emphasizing lateral communication. Therefore, according to Burns and Stalker, the organizational structure should be characterized by low vertical differentiation, low formalization, and decentralization (Hendrick, 1986). Authority should be based on expertise not hierarchical level. 2.6 Organizational Design The organizational design can be defined by three components: complexity, formalization, and centralization (Hendrick, 1986). Complexity describes the degree of differentiation and integration within an organization. The three types of differentiation include horizontal the degree of job specialization or the separation between units, vertical the depth of the hierarchy within the organization, and spatial dispersion the degree to which the organizations personnel and facilities are geographically dispersed. An increase in any type of differentiation results in an increase in the organizations complexity level. Complexity is managed by integrating mechanisms, which are designed into the organizations structure to facilitate communication, coordination, and control throughout the organization (Hendrick, 1986). Some examples of integrating mechanisms include formal rules and procedures, committees, liaisons, and information and decision support services. There is a direct relationship between the integrating mechanisms and the degree of differentiation; as differentiation increases, the integrating mechanisms must increase. Formalization describes the degree to which jobs are standardized in an organization. When jobs are highly formalized, the employee has little control over what is to be accomplished and the manner in which to accomplish the task. When jobs have little formalization, employees can use their discretion about what is to be accomplished and how to accomplish the task. Centralization is the degree to which formal decision-making is allowed at the individual, unit, or a higher level (Hendrick, 1986). In decentralized organizations, decision-making is accomplished at the lowest level possible. In highly centralized organizations, employees have little input into the decisions affecting their jobs. In highly unstable environments, decentralization is recommended because decisionmaking that affects an individuals job provides a source of motivation and commitment (Hendrick, 1986). Groups and teams are a trend in industry for product design due to advances in technology and a global economy (Salas et al., 1992; Bursic, 1992; Bowers et al., 1997). This trend includes the domain of engineering design. Time to market and maintaining quality at low costs are factors driving the use of 14 teams for engineering design (Frankenberger et al., 1997). The use of teams instead of individuals increases the degree of complexity because additional relationships must be maintained and coordinating mechanisms must be added. According to McGrath (1984), a group is a social aggregate of individuals who are mutually aware of group members and amongst whom there is the potential for interaction. Many researchers define a group based on the ability of the members ability to distinguish themselves from the nonmembers (Walton & Hackman, 1986; Brett & Rognes, 1986). Groups also rely on independent interaction and can be successful with or without the full participation of all members (Hall & Rizzo, 1975). The definition of a team is more complex. One definition of a team is a distinguishable set of two or more people who interact dynamically, interdependently, and adaptively toward a common and valued goal/objective/mission, who have each been assigned specific roles or functions to perform, and who have a limited life-span of membership (p. 4). (Salas et al., 1992) The themes underlying this definition of a team include coordination, information and resource sharing, adapting to task demands, and the existence of an organizational structure, all contributing to completing the task. Fleishman and Zaccaro (1992) identified two characteristics of teams include a shared task orientation and task interdependency. Interdependency among members appears to be a key distinguishing factor between a team and a group; team members are interdependent (Salas et al., 1992; Fleishman et al., 1992; Shiflett, Eisner, Price, & Schemmer, 1985). For this study, Orasanu and Salas (1993) perspective of a team was particularly applicable. They defined a team to be individuals working together at small tasks within the scope of a larger task, while contributing personal expertise and knowledge, often under intense workloads accompanied by high stress factors (p 556). (Orasanu et al., 1993) This definition of a team is relevant to this study since one of the goals was to determine the influence of team process on individual and team workload levels. 2.6.1 Engineering Design Teams Design is the responsibility of both groups and individuals. Rarely is design completed in isolation. Often distinguishing a single individual as the designer is difficult because design has become a team activity (Taylor, 1993). Designers continuously exchange information even if working on a design individually (Ehrlenspiel et al., 1993). One of the problems with design processes pointed out by Ehrlenspiel (1993) was that many of the early investigations on design were based on individuals working in complete isolation on a design problem. Those early observations were then used to formulate the engineering design process. While groups tend to be the norm in engineering design, a useful purpose for components of the engineering design process to be conducted as individuals working alone or in parallel may still exist. Anecdotally, individuals have been found to generate the most creative ideas, while idea execution required a group (Holt, 1993). While the trend has been towards using groups and teams in engineering design, little research has been conducted on this approach to engineering design. The lack of research is surprising since the existing group literature indicates conflicting support in favor of group work. Several studies, conducted in the 1960s, invited skepticism on the usefulness of teamwork (Briggs et al., 1965; Naylor et al., 1965; Kidd, 1961). Through his research, Kidd (1961) found that performance did not improve as group size increased when comparing individuals, dyads, and triads. However, Kidd did acknowledge that group efforts might be beneficial in the long run when considering the intrinsic motivation generated by individuals working in groups. Supporters of collaborative engineering design argued the benefits include an increased knowledge base and potential cost savings, which reduce the potential for error after the design phase has technically concluded (Kidd, 1994). 15 As mentioned previously, much of the early research on engineering design consisted of observational studies. For example Olson, Olson, Carter, and Storrsten (1992) observed ten software design groups in two organizations during the initial design phase. The goal was to observe groups working on software design. Similarities between all groups included designers clearly stated issues, alternatives, and criteria for alternative evaluation, planned their activities, and shared their expertise. In addition, design discussions and patterns of activity were similar. Designers spent 40% of the time on design discussion and 30% of the time on summaries and walkthroughs. Meetings seemed to contain large amounts of clarification. In Olsons (1992) study of thirty-eight collocated design groups with computer collaboration support versus unsupported groups, they found supported groups tended to produce higher quality designs than unsupported groups. However, the supported groups did not spend as much time exploring the design space as unsupported groups. More recently, the collaborative engineering design process has been studied in controlled experimental settings. Meredith (1997) conducted research on collocated engineering design teams. He studied the effect of engineering methodology (sequential versus concurrent), group size (three and six members), and level of computer supported cooperative work on design performance, satisfaction, lifecycle cost, process cost, and process time. Meredith found design performance was not affected by the engineering methodology or the use of groupware to support the conceptual design process. However, design performance of small groups was significantly better than design performance of large groups. Large groups incurred significantly higher life-cycle costs and process costs compared with small groups. As expected, the process cost for computer-supported work was higher than the process cost for unsupported groups. One result that contradicted previous group research was the similarity in process time between large and small groups. The literature indicated that groups tend to use coordination and communication processes to come to agreements on decisions. Therefore, as group size increases, the difficulty involved with managing coordination should increase thus increasing process time (Shaw, 1976; Hare, 1981; Hill, 1982). Satisfaction was not affected by the treatment. Based on Merediths (1997) research, there are implications for how structured support for the group process and group size will affect the design process. For example, performance measures were similar regardless of group size and structured-computerized group support; however, there were significant cost implications. His results indicated that groups of three were sufficient to study the engineering design process and resulted in lower life-cycle and process costs. Furthermore, Merediths results indicated that the cost of providing a structured computer tool (groupware) during the conceptual design phase was not justified due to a lack of return on performance for collocated engineering design groups with three and six members. A study that extended the study of engineering design to distributed groups was conducted by Harvey (1997). Distributed collaborative design groups were explored as the first step towards the development of a model for understanding the distributed engineering design process. Specifically, Harvey was interested in the affect of task complexity (controlled at a high level of complexity) and collaborative technology (in the form of communication media and conversational props) on individual and group development (in the form of vocabulary schema and individual cognitive resources) during a three-phase approach to engineering design. Communication media included collocated communication, audio and video/audio communication; conversational props included the design support computer tools, for example, word processing, spreadsheets, and CAD. Vocabulary schema referred to the type of words used throughout the design process. The three measures of vocabulary schema included the number of words, number of different words, and number of unique words over five minute intervals. To study the difference between group and individual development, he measured task cohesion, cognitive resources, and vocabulary schema. The main focus of Harveys research was to study the process and purposefully did not address outcome measures associated with the design process. From Harveys study, there was an indication that group development was slower for audio groups compared with face-to-face groups. In addition, he found the richness of communication props 16 used by the team decreased over all design phases in the audio groups during the visualization phase. Based on the study of design occurring naturally, this is a commonly observed behavior regardless of the communication medium (Ball et al., 1994). To help explain the observations made during the design process, Harvey used the NASA Task Load Index (NASA-TLX) to study subjective workload levels over time. He found significant differences in the workload associated with the difference communication media during the first phase of engineering design (referred to as the conceptualization phase). Audio/video groups reported a significantly higher workload than either collocated or audio groups. From Harveys (1997) research, there were implications that arose from the study of collaborative engineering design in a distributed environment. For example, there was support that different communication media might be better suited for certain tasks within the design process; similarly, the required level of design support appeared to vary throughout the design process. In addition, there was evidence that communication media contributed to the designers workload level. However, based on the experimental design, questions remain as to whether or not the workload resulted strictly from the communication technology used or was it due in part to the nature of the team functioning and/or design support tools. Furthermore, the changes in workload over time were not evaluated. 2.6.2 Project Teams Project teams tend to be a special case of the traditional team. Many project teams are formed with for specific periods of time and once the project is complete, the team disbands. A common trend among project team members is a lack of familiarity with one another. However, the tasks involved with the project are interrelated and required coordination, information and resource sharing. The members of a project team tend to have a wide variety of backgrounds and are part of a team for a relatively short period of time (Pinto et al., 1991). Cooperation is believed to contribute to successful project completion (Pinto et al., 1991; Pinto et al., 1993). However, when individuals from diverse professional backgrounds work on a project, cooperation has been difficult to achieve. From a survey of project team members working in a hospital environment (273 respondents), accessibility, ground rules and procedures, and a clear project goal all were significantly correlated with cooperation on project teams. The basic assumption underlying the use of teams has recently attracted some attention in the project management literature. A series of studies were conducted to explore the planning phase within project management (Kernaghan et al., 1986; 1990). Individuals who were placed into nominal and interacting groups arranged a series of planning activities in the most logical order. The outcome was compared to expert opinion and a score was given to each individual and group. Interacting groups, that is groups that discussed the order of activities, had higher scores than the average individual score. However, groups did not have higher scores than the best individual. Because this was an activity ordering exercise, extending the results to actual planning activities, like scheduling and budget development, is difficult. 2.6.3 Groups versus Individuals The concept of individuals versus groups has been widely explored in the literature; however, many of the results are contradictory. The presence of another person often changes an individuals performance. Sometimes performance was enhanced through competition and stimulation caused by anothers presence. Other times performance degraded due to distractions, group norms, limited participation, resistance to task work, and group think (Hare, 1976; Gustafson, Shukla, Delbecq, & Walster, 1973). Individuals have been found to generate more ideas and unique solutions for problems than groups (Rotter et al., 1969). Furthermore the time and effort required for maintaining the group was found to exceed the time and effort involved with individual work (Gustafson et al., 1973). The 17 individual is typically considered superior to groups based solely on the number of hours required to complete a task (Barker et al., 1991; Hare, 1992). Frequently groups are used based on the assumption that the social motivation provided by teams will be sufficient to enhance performance compared to an individual working in isolation (Hackman et al., 1975). Anecdotal support for teams showed that individuals resources can be pooled to improve performance (Johnson & Torcivia, 1967; Smith, 1989). Learning may also be facilitated through the team members interaction (Tuckman & Lorge, 1965). And as Hackman and Morris (1975) indicated, there are instances where teams may be preferred due to large workloads. One of the explanations for negative outcomes with group work is the additional effort required for coordination and communication (Steiner, 1972; Beith, 1987; Bowers et al., 1997). This additional effort is referred to as process loss (Steiner, 1972). In one study on software design groups, groups spent 20% of their time on coordination alone (Olson et al., 1992). However there is evidence that as task complexity increased, the losses due to coordination and communication are offset such that group performance was superior to individual performance (Davis & Harless, 1996). This was the case in a two-stage price-searching task. When complex searching methods with multiple iterations of decisions were required, groups were superior to individuals. However, when less complex searching was required with fewer iterations, individuals were superior to groups (Harless et al., 1996). Research on individuals versus groups is often approached based on the type of task. One taxonomy divides problem-solving tasks into disjunctive and conjunctive tasks (Smith, 1989; Steiner, 1972). Disjunctive problems include those tasks where success is achieved if a single person in the group has the correct solution. Conjunctive problems are those tasks where success must be achieved through the combination of information and coordination among group members (for example, design). Research has found that small groups performed better than individuals for tasks that are unitary and disjunctive (Tuckman et al., 1965; Davis et al., 1996). However, when comparing the group to the best performer of a nominal group, groups tend to have lower performance due to process loss (Davis et al., 1996). Other research has found small groups performed better than individuals for tasks that were multistage and conjunctive (Smith, 1989). After studying both conjunctive and disjunctive tasks over a 10-year period, Smith (1989) found real groups performed significantly better on task grades and independent examination grades than nominal groups for conjunctive tasks. No differences between real and nominal group performance was found for disjunctive tasks. Real group performance was significantly better as the number of problem stages increased compared to nominal group performance. Motivation was also found to be higher in real groups compared with nominal groups, which supports the concept that the presence of other individuals is a source of stimulation. For decision-making tasks the results are also mixed. Some research has found that there are no differences in outcomes between individuals and groups (Davis & Toseland, 1987). Groups have also been found to make poorer decisions than individuals (Hare, 1976; Gustafson et al., 1973). Similarly, some research indicated groups made better decisions (Hill, 1982). Orasanu and Salas (1993) clarified the difference between teams and individuals for decision-making; teams have a combined source of information and perspective from which a decision can be made. While the results are mixed for task performance based on quality of decision or accuracy of task; there is generally support that groups take significantly longer to complete tasks compared to individuals (Hill, 1982; McGrath, 1984). Other outcomes have included a measure of satisfaction. Some research indicated satisfaction was not affected by working in groups compared to working alone (Davis et al., 1987). The literature guiding the use of individuals or groups tends to be mixed. For example according to Hare (1976), groups are superior to individuals on manual tasks compared to intellectual task. In addition, groups should be more productive than individuals if there is a need for division of labor, the 18 task is easily controlled, and the group maintains a high standard of productivity compared to the individual. Baker (1991) also provided guidance on the choice of individuals to groups. He recommended that groups are preferable to individuals when the participants have a common understanding and information base, the environment supports the need for a variety of points of view, groups have the ability to create more ideas with greater variety than individuals, action will taken, and creativity can be enhanced through the exchange of ideas. 2.6.4 Group Size Group size is an additional factor that might have implications on performance. O'Reilly and Roberts (1977) found that group size was positively related to horizontal and vertical differentiation, such that as group size increased, horizontal and vertical differentiation increased. Furthermore they found that communication was negatively impacted by an increase in group size. Other research has supported that size has a negative impact on performance (Nieva et al., 1985). Group process and outcomes are another area affected by the groups size (Dennis, Nunamaker, & Vogel, 1990). As group size increased, the time available per member for interaction decreased resulting in fewer opportunities for each member to participate (Shaw, 1976). In addition, with larger groups there was a tendency for a few members to dominate the discussion, there was less exploration of viewpoints, and more direct attempts to reach a solution were made (Hare, 1981). Previous research on group size found that groups of three or more members resulted in significantly different performance than individuals (Holloman & Hendrick, 1971; Smith, 1989; Rotter et al., 1969). Hackman and Vidmar (1970) made pair-wise comparisons between groups of 2, 3, 4, 5, 6, and 7 members (for example, dyads were compared to triads and triads were compared with dyads and groups of four). They found the difference in performance between dyads and triads was twice the difference in any other comparison for thirteen of twenty-seven performance measures. Furthermore, for eleven of the thirteen measures, the difference between dyads and all other group sizes were significantly different. While performance appeared the highest for teams of four to five members, satisfaction was higher in smaller teams (Hackman et al., 1970). In addition, from Merediths (1997) study of the effect of group size on the performance of engineering design groups, groups of three had significantly higher design performance and lower life-cycle and process costs than groups of six. Hoffman (1979) found that the performance of groups of three tended to be generalizable to larger groups. The results from research conducted on dyads may not generalize to larger sized groups. Studies have also shown that as groups become large, the members tend to work in smaller units of two to three members within the larger group (Holloman et al., 1971). The group size should be small enough that all members have the opportunity to participate, but big enough to include all the people with the appropriate skills required for the problems solution (Barker et al., 1991). 2.6.5 Group Process Defining effective group process behaviors has been historically difficult. Contributing to the problem of measuring group process is that over 130 different labels exist to describe effective team functioning (Salas et al., 2000). Group process is the interaction that occurs between team members in order to complete a team task (Davis, Gaddy, & Turner, 1985). There are two distinguishable categories of work that occur during the teams life-cycle: task work and teamwork (Morgan et al., 1986; Baker et al., 1992). Task work includes behaviors related to the tasks to be performed by the individual members. Critical behaviors are those related to the individual executing individual member functions. Teamwork is related to interaction between members. Seven critical behaviors related to a teams success or failure were identified using military teams. These behaviors include communication, coordination, cooperation, ability to give suggestions and criticisms, ability to receive suggestions and criticisms, team spirit and 19 morale, and adaptability (Morgan et al., 1986). Effective teams were distinguished from ineffective by the occurrence of these behaviors (Oser et al., 1989; Morgan et al., 1986; Brannick et al., 1993). 2.6.5.1 Communication Communication is the exchange of information between a sender and receiver(s) (McIntyre et al., 1995; Cannon-Bowers et al., 1995) for the purpose of clearly and accurately sending and acknowledging information, instructions, or commands (Brannick et al., 1995; Cannon-Bowers et al., 1995). Included in the information exchange are asking, answering, informing, and directing strategies (Davis et al., 1985). Communication should be closed loop such that a sender initiates the message; a receiver accepts the message and provides feedback to indicate reception; and the sender double checks the meaning was understood (Cannon-Bowers et al., 1995; Swezey et al., 1992). Some of the purposes of communication include information exchange, arriving at a common understanding between team members, strategizing potential solutions, and defining team members responsibilities involved with the task (Orasanu, 1990; Bowers et al., 1992). Effective communication skills have been correlated with higher performance (OReilly et al., 1977; Davis et al., 1985; Oser et al., 1989; Brannick et al., 1993; McIntyre et al., 1995). Most research in this area was conducted using military teams. Providing feedback is one aspect of communication correlated with successful team performance. Praising each other, making positive statements (Oser et al., 1989), asking for help when needed (McCallum et al., 1989), asking for specific clarification on a communication that was unclear (Morgan et al., 1986; Baker et al., 1992; Prince et al., 1992; Oser et al., 1989), and acknowledging team members speech (Brannick et al., 1995) was positively related to successful performance. When aircrews were allowed to provide feedback to each other, errors were found to be less frequent (Johnston et al., 1968). Other characteristics of communication that successful teams have exhibited include asking relatively fewer questions, providing fewer answers in the form of a question, and providing fewer responses to questions (Urban et al., 1995; Urban et al., 1993). Existing teams consistently exhibited stronger communication skills compared to novice teams (Brannick et al., 1995). Early research indicated that communication might be detrimental to team output (Johnston, 1966; Williges et al., 1966; Naylor et al., 1965). However, the majority of this research involved communication during highly structured aviation tasks. For relatively unstructured tasks, a positive relationship existed between the amount of communication and performance when performance was measured based on quantity and quality (Nieva et al., 1985). Some aspects of communication have been found to be negatively associated with performance. One example was unnecessary interruptions between members led to lower performance (or failure) (Foushee, 1984). Ineffective teams tended to ask more questions and provide more responses to address information about the resources available to complete their task. Conversely, effective teams were better at anticipating each others needs (Urban et al., 1993). Effective teams acknowledged communication (Stout et al., 1994; Prince et al., 1992), provided information when asked (Stout et al., 1994), used proper terminology when communicating information to others (Morgan et al., 1986; Prince et al., 1992), double checked each others work, offered feedback in a non-threatening manner, communicated information in proper sequence, and accepted constructive criticism appropriately (Morgan et al., 1986). When mistakes were made, the member observing the mistake discussed the mistakes privately with the member in question (Morgan et al., 1986). Communication has been demonstrated to mediate the effects of workload and structure. The amounts of communication changed under various workload conditions (Kleinman, 1990). Causal communication among members during periods of low workload improved overall performance for some teams (Helmreich, 1984). Under high workloads, better teams asked more questions and answered more questions (Urban et al., 1993). Under higher workloads effective teams were also able to anticipate each 20 others needs (Kleinman, 1990; Urban et al., 1993). In another study, as workload increased, members neglected more and more obligations. The burden of initiating communication was placed on the user of the information, not the immediate source of the information (Roby & Lanzetta, 1957; Lanzetta & Roby, 1956; Banderet, Stokes, Francesconi, & Kowal, 1980; Banderet, Stokes, Francesconi, Kowal, & Naitoh, 1981; Dyer, 1984). 2.6.5.2 Cooperation Cooperation involves members knowing the strengths and weaknesses of each team member, the willingness to assist members when needed, and the ability to change the work pace to fit the teams requirements (Cannon-Bowers et al., 1995; Alexander et al., 1965 as cited in Baker et al., 1992). Some examples of cooperation include prompting another member on what to do next and assisting another member when help is needed (Morgan et al., 1986; Baker et al., 1992; Oser et al., 1989; McIntyre et al., 1995). In addition, members in need of assistance willingly accepted help. 2.6.5.3 Coordination Coordination is the integration or linking of different components of an organization for the purpose of accomplishing a set of goals and objectives (Dailey, 1980). In other words, coordination is a process that brings together resources and activities to complete a task in the time allowed (CannonBowers et al., 1995; Davis et al., 1985). Coordination occurs in the sequencing of tasks, the integration of tasks, and the timing and pacing of activities (Cannon-Bowers et al., 1995; Dyer, 1984; Dailey 1980). Members provide direction on what should be completed next and adjust their task work to that of the other members (Morgan et al., 1986; Baker et al., 1992; Brannick et al., 1993). Technical coordination refers to accomplishing a set of actions in the proper sequence and providing useful information when appropriate (Brannick et al., 1993; Cannon-Bowers et al., 1995). Interpersonal coordination is the quality of team members interchanges, for example, arguing or encouraging (Brannick et al., 1993; Cannon-Bowers et al., 1995). Coordination includes the ability to anticipate team members needs and communicate efficiently (Cannon-Bowers et al., 1995). Well-coordinated teams have been observed to move easily between tasks, obtain information from other team members when necessary to complete a task, consult with members if there is uncertainty about the next step, anticipate and provide other members informational needs (Morgan et al., 1986). Effective teams have also been observed to help each other when needed, structure information gathering, learn about or assist others with their tasks when not busy with own task (Oser et al., 1989; Morgan et al., 1986). Coordination is closely tied with communication. Easy access to needed information was found to facilitate performance. Increased access to information through communications with each team member and monitoring members enhanced performance under high workloads. Increased workloads negatively affected individual responsibilities and responsibility to the team resulting in an overall reduction in performance (Moore, 1962). Under high workloads, operationalized by the number of resource transfers, coordination became implicit, but under lower workloads transfer of information tended to occur only after specific requests (Kleinman, 1990). With increasing workloads, operationalized as the number of information inputs, performance decreased (Johnston et al., 1968; Kidd, 1961; Lanzetta et al., 1956). Kidd (1961) varied team size between individuals, dyads, and triads. As workload increased, performance decreased. When load was constant, as team size increased, team performance moderately improved. When load increased in proportion to size, performance diminished in multiple person teams. He concluded that performance was the best when coordination demands were minimal. 21 Closely related to the concept of coordination are organization and orientation, which have also been identified as skills for effective team performance (Baker et al., 1992; Hogan et al., 1989; Fleishman et al., 1992; Dyer, 1984; Lanzetta & Roby, 1960; Shiflett et al., 1985). Organization refers to how tasks and activities are structured (Hogan et al., 1989). Included is task sequencing, activity pacing, response orders, assigning priorities to activities, balancing the load, and matching members skills to tasks (Nieva et al., 1978; Dyer, 1984; Fleishman et al., 1992; Shiflett et al., 1985). Many of these concepts are integral to project management tools and techniques. Orientation is the generation and distribution of information about team goals, team tasks, and member resources and constraints (Nieva et al., 1978 as reported in Dyer, 1984; Shiflett et al., 1985). In a study conducted by Hackman, Brousseau, and Weiss (1976), groups were provided with either equal or unequal information and one of three orientation methods: encouraging discussion on how to accomplish task, discouraging discussion by immediately working, and not providing any directions. Little discussion resulted when instructions were not provided. When group members were provided with unequal information, encouraging discussion improved group effectiveness. However, when group members had equal information, no discussion resulted in higher group performance. Discussion did not occur in the control groups (those provided with no direction). While, groups that had discussions were more flexible, however, they had more task and interpersonal conflicts. One observation was that discussion improved performance for emerging tasks but not established tasks. In a similar study conducted by Shure, Rogers Larsen and Tassone (1962), groups were put into one of three conditions, no opportunity for discussion, an opportunity to plan during the period of task completion, or an opportunity to plan between periods of task completion. Only those groups that were able to plan between work periods performed well. Effective teams exchanged information about member resources and constraints, team task and mission/goals, environmental characteristics and constraints, and assigned priorities among tasks (Nieva et al., 1978; Fleishman et al., 1992; Shiflett et al., 1985). Effective members were able to state the purpose of the team mission, describe which team member will satisfy the tasks to execute the mission, and describe the general approach to accomplish the mission (Swezey et al., 1992). Furthermore, effective teams had clear and objective team goals (Dyer, 1984; Hall et al., 1975; Swezey et al., 1992) and there was consistency between team and individual goals (Ilgen, Shapiro, Salas, & Weiss, 1987; Swezey et al., 1992). 2.6.5.4 Giving Suggestions/Criticisms The giving of suggestions and criticisms refers to the ability of team members to provide their colleagues with feedback and recommendations. Effective teams suggested ways to find errors (Oser et al., 1989; Brannick et al., 1993). In addition, effective teams found ways to provide suggestions and criticisms in a non-threatening way (Brannick et al., 1993). 2.6.5.5 Team Spirit and Morale Team spirit and morale is closely related to cohesion. It includes how comfortable the team members are with each other, the presence of hostility or boredom. Also included is whether or not the team discusses ways to improve their performance as a team (Morgan et al., 1986; Baker et al., 1992). Team spirit and morale is also similar to the concept of motivation, which has been identified as a distinguishing factor of effective teams (Shiflett et al., 1985; Nieva et al., 1978; Baker et al., 1992; Fleishman et al., 1992; Ilgen et al., 1987; Swezey et al., 1992; Morgan et al., 1986; McIntyre, Morgan, Salas, & Glickman, 1988). Motivation refers to the teams ability to define team objectives related to the task and energize the team towards these objectives. This includes developing performance norms and reinforcing task orientation (Shiflett et al., 1985). Effective teams showed signs of mutual support 22 (Morgan et al., 1986; Swezey et al., 1992). For example, teams made motivating statements and praised team members (Oser et al., 1989; Brannick et al., 1993). Other characteristics of effective teams included the development and acceptance of team performance norms, linking rewards to achieving team performance, reinforcing task orientation, balancing team orientation with individual competition, and resolving performance-relevant conflicts (Nieva et al., 1978; Fleishman et al., 1992). Members that did not feel central to team success were not as satisfied as members who were central (McIntyre et al., 1988; Swezey et al., 1992). Effective teams were supportive of team members even when members made a mistake (Morgan et al., 1986; Swezey et al., 1992) and made positive statements to motivate the team (Oser et al., 1989; Swezey et al., 1992). Effective teams had rules of behavior agreed on by all members (Hackman, 1987 cited in Sundstrom, De Meuse, & Futrell, 1990). The role of group norms is to establish high standards of performance then cohesion can positively affect performance (based on Schachter, Ellertson, McBride, & Gregory, 1951). This has been supported by several studies (Berkowitz, 1954, Schachter et al., 1951; Seashore, 1977; Fleishman et al., 1992). 2.6.5.6 Acceptance of Suggestions/Criticisms Acceptance of suggestions and criticisms refers to a team members ability to take criticisms and suggestions and learn from this type of feedback to improve his or her performance. Effective teams were distinguished from ineffective teams through thanking members for pointing out mistakes (Oser et al., 1989; Brannick et al., 1993; Morgan et al., 1986; Baker et al., 1992). 2.6.5.7 Adaptability Adaptability, or flexibility, is a teams ability to alter a course of action when conditions change and maintain constructive behavior under pressure (Brannick et al., 1995). It is a process the team goes through to use information from the environment to develop and adjust strategies through compensating, timing, correcting errors, and reallocating team resources (Cannon-Bowers et al., 1995; Shiflett et al., 1985). Team members helping others who were having difficulties was a distinguishing factor between more and less effective teams (Oser et al., 1989; Brannick et al., 1993). Effective team members changed the way they performed a task when asked (Morgan et al., 1986; Baker et al., 1992; Swezey et al., 1992), were responsive to feedback and willing changed direction in the plan, strategy, and organization (Hogan et al., 1989), altered behavior to meet situational demands, helped other members (Stout et al., 1994; Prince et al., 1992; Cannon-Bowers et al., 1995), provided mutual critical evaluation, correction of error, and mutual compensatory timing (Shiflett et al., 1985; Fleishman et al., 1992; Cannon-Bowers et al., 1995), and adapted to relevant information (Morgan et al., 1986; Swezey et al., 1992). 2.6.5.8 Other Effective Group Behaviors Cohesion, leadership, decision-making, and assertiveness in addition to members having shared objectives and clear roles and responsibilities have also been identified as indicators of effective teamwork (cohesion: Sundstrom et al., 1990; Cartwright, 1968; Fleishman et al., 1992; Hare, 1976; Nieva et al., 1978; Driskell et al., 1992; Zaccaro et al., 1988; Swezey et al., 1992; Brannick et al., 1993; Bowers et al., 1992; Daily, 1980; leadership: Blickensderfer et al., 2000; Brannick et al., 1995; Sundstrom et al., 1990; McGrath 1984; Stout et al., 1994; Prince et al., 1992; Swezey et al., 1992; Franz et al., 1990; Cannon-Bowers et al., 1995; Decision-making: Blickensderfer et al., 2000; Brannick et al., 1995; Stout et al., 1994; Prince et al., 1992; Gist, Locke, & Taylor, 1987; Cannon-Bowers et al., 1995; Assertiveness: Blickensderfer et al., 2000; Brannick et al., 1995; Stout et al., 1994; Prince et al., 1992; Shared objectives: 23 Tjosvold & Johnson, 2000; Salas et al., 1992; Clear roles and responsibility: Neck et al., 2000; Orasanu et al., 1993). A measure of the members desire to remain within a team (or group) is referred to as cohesion and is related to mutual attraction and shared beliefs and coordinated behaviors amongst members (Cartwright, 1968; Fleishman et al., 1992; Driskell et al., 1992; Daily, 1980). Fleishman (1992) observed that research on cohesion has found that it has mixed effects on performance (positive: Hare, 1976 14 studies; Lott & Lott, 1965 20 studies; Stogdill, 1972 12 studies; and Nieva et al., 1978 8 studies) (negative or zero: Lott et al., 1965 15 studies; Stogdill, 1972 20 studies; Nieva et al., 1978 6 studies). One study providing insight on the mixed results was conducted by Seashore (1954). In Seashores study, he found that group norms acted as a moderator between cohesiveness and performance. More cohesive groups were able to set and meet their group goals better than less cohesive groups. Depending on the group goals, performance was affected positively or negatively. Cohesion has been correlated with performance, coordination (Daily, 1980; Brannick et al., 1993), group interaction, influence, and productivity (Shaw, 1976). When task performance was based on the sum of individual efforts (additive task), task-based cohesion positively correlated with performance (Zaccaro et al., 1988; Fleishman et al., 1992). A lack of cohesion caused by production errors was related to later deficiencies in team coordination (Foushee, 1984; Swezey et al., 1992). Cohesion was influenced by a members preference to work as an individual or on a team (Driskell et al., 1992). With respect to workload, good performers had higher levels of cohesion under high workloads (Bowers et al., 1992). Leadership is the direction and coordination of others activities and performance monitoring, task assignment, and motivation of team members (Brannick et al., 1995; Cannon-Bowers et al., 1995). Effective leaders provided direction, organization, and guidance on activities (Hogan et al., 1989), assigned tasks, and provided feedback on team and individual performance (Stout et al., 1994; Prince et al., 1992). Leaders verbalized their plans for achieving team goals (Helmreich, 1982 as cited in Swezey et al., 1992), kept the team focused on matters at hand, asked for input, and discussed potential problems (Franz et al 1990 as cited in Swezey et al., 1992). Decision-making is the use of sound judgment to select the best course of action based on available information. Decision-making includes searching for information, allocating and monitoring resources (Brannick et al., 1995; Cannon-Bowers et al., 1995). In a team context, an emphasis is placed on the skill of pooling information and resources in support of a choice (Cannon-Bowers et al., 1995). In comparing existing teams to novice teams, the experienced team showed superior performance in decision-making (Brannick et al., 1995). Effective teams assessed the problem (Cannon-Bowers et al., 1995), identified alternatives and contingencies (Prince et al., 1992; Gist et al., 1987; Cannon-Bowers et al., 1995), gathered data prior to making a decision (Stout et al., 1994; Prince et al., 1992), evaluated alternatives (Gist et al., 1987; Cannon-Bowers et al., 1995), and provided a rationale for the decision (Stout et al., 1994). In addition, participation and consensus building were encouraged in effective teams (Gist et al., 1987). Assertiveness is an individuals willingness to make decisions and act on them, to defend a decision, and to admit ignorance and ask questions (Brannick et al., 1995). In comparing existing teams to novice teams, members of the existing teams were more assertive than the novice teams members (Brannick et al., 1995). Assertive individuals made suggestions, maintained a position when challenged (Stout et al., 1994; Prince et al., 1992), and asked questions when uncertain (Prince et al., 1992). Several non-observational measures exist in the literature, which attempt to assess team effectiveness. These measures include monetary value or the dollar value of the team effort; performance time or the time to complete the task; productivity, criterion-referenced evaluation and quality of work output (Hogan et al., 1989). 24 Many of the concepts discussed as effective group behaviors are also related to job satisfaction. Some specific examples include clearly defined responsibilities and access to needed information. In addition, several of the recommendations from the literature were incorporated into the design of the experiment. For example, participants were asked to develop a goal for the task; all participants were trained in all functions of the problem. 2.6.6 Job Satisfaction The study of job attitudes did not become important until the 1930s, but currently is a heavily studied construct. Job satisfaction has been defined as a positive or enjoyable emotional response that comes from an individuals self-evaluation of his or her job or job experience (Locke, 1976). There are several underlying theories in the approach to understanding and measuring job satisfaction, which can be divided into two categories (Jewell, 1998). The first category consists of dispositional (or need-based) theories, for example Maslows need hierarchy and Herzbergs motivator-hygiene theory. The second category of job satisfaction consists of cognitive theories, for example, Vrooms expectancy theory and Lockes goal setting theory. In Maslows (1943) hierarchy of needs theory, humans have five requirements: psychological needs, safety needs, social needs, esteem needs, and self-actualization needs. Under this theory, a need is never fully met. However, once a need has been substantially satisfied, acquiring that need is no longer a motivator. The motivation-hygiene theory, another need-based theory, proposed that job satisfaction is affected by intrinsic and extrinsic factors (Herzberg, 1966). An example of an intrinsic factor includes the opportunity for personal growth on the job. A financial reward is an example of an extrinsic factor. Contributing to this theory were the findings from a set of interviews from two hundred accountants and engineers. Participants were asked about work events that contributed to improvements or reductions to their perceptions of job satisfaction. The five factors that were most commonly referred as motivators included achievement, recognition, work itself, responsibility, and advancement. The most common factors that were detractors included company policy and administration, supervision, salary, interpersonal relations, and work conditions. In general, according to the need-based theories, individuals will do certain things in order to satisfy specific needs. The value from these theories is an understanding in how work characteristics might satisfy an individuals wants and needs. Generally, cognitive theories describe the psychological process that is involved in making decisions. Cognitive theories have roots in the 1930s, but did not become a significant topic for research until the 1960s. The cognitive theories have a strong empirical research base (Jewell, 1998). A cognitive theory that has contributed to the understanding of job satisfaction is expectancy theory (Vroom, 1982). Expectancy is the association between an action and the outcome (Vroom, 1982). Based on this theory, the extent to which an individual exhibits an anticipated behavior depends on the degree to which the outcome is attractive to the individual. For example, if rewards are provided for individual behaviors, the expectation is that individual behaviors are the preferred behavior as opposed to group and cooperative behaviors. Another cognitive theory is Lockes (1968) goal-setting theory. In this theory, the act of setting a goal increases an individuals motivation because the simple act of accomplishing a goal is rewarding. Based on research, there are five guidelines to successful goal setting (Jewell, 1998; Locke, 1968). First, the goal should be specific. Second, the goal should not be too easily accomplished. Third, the person who is charged with accomplishing the goal must accept the goal. Fourth, individuals need to receive feedback on their progress towards achieving their goal. And fifth, participation in setting goals might be better than goals that are assigned. 25 Previous studies on job satisfaction have found positive correlations between stressors, including workload, anxiety, frustration, and job dissatisfaction (Spector, Dwyer, & Jex, 1988; Jex & Gudanowski, 1992). The primary source of the correlations has been self-reports. Studies on job satisfaction have found mixed results that being a member of a small cohesive group will increase job satisfaction (Argyle, 1992). Many of the studies with this finding used students or newly formed groups without a previous history. Job satisfaction has been linked to workload. If there is too little work and/or challenge, then the worker will be expected to experience boredom as a result of unused mental capacity. However, if there is too much challenge or work, then the mental capacity might be overwhelmed. In either case, the expected result is dissatisfaction (Locke, 1976). Some studies have explored the relationship between performance and job satisfaction. In a meta-analysis conducted by Petty, McGee and Cavender (1984), fifteen studies that included measures of both job satisfaction and individual performance from a 1964 review by Vroom were analyzed. The studies that were included in the meta-analysis were limited to those studies that used the job descriptive index (JDI) to measure job satisfaction, thus limiting the generalizability of the results. The average correlation between job satisfaction and individual performance was 0.14. In an analysis of twenty studies from the 1970s and 1980s, the correlation between job satisfaction and individual performance was 0.31 for professions and 0.15 for nonprofessionals. A more extensive meta-analysis was conducted by Iaffaldano and Muchinsky (1985). In this study, seventy articles published between 1944 and 1983 were reviewed. The correlation between job satisfaction and performance was 0.146. Generally, the relationship between job satisfaction and individual performance has been reported to be a small, positive correlation. 2.7 Performance Measures To understand how the system is performing, performance measures were collected and analyzed for each of the sociotechnical subsystems including the technological and personnel subsystem and the organizational design. 2.7.1 Technological Subsystem Performance Measures Within the technological subsystem, performance was assessed using two different measures: design performance and planning and tracking performance. 2.7.1.1 Design Performance Design performance was defined by cost-effectiveness (Blanchard et al., 1990; Meredith, 1997). Cost effectiveness is a first-order parameter comprised of two second-order parameters: life-cycle cost and system effectiveness (Blanchard et al., 1990). Also associated with design performance is design cycle time, which is the sum of the process times for individual design phases. Process time only included task related work. Therefore, the time involved with completing questionnaires and forms were not included in process time. Both design cycle time and time in each design phase were evaluated. 2.7.1.2 Planning and Tracking Performance Planning performance was defined based on the deliverables ability to meet specifications, the difference between the estimated and actual costs, and the difference between the estimated and actual time to complete the project. These differences are referred to as cost variance and schedule variance. The variances were converted into a ratio, called the cost performance index and the schedule performance index, to facilitate interpreting performance (Gido et al., 2000). When the ratios less than 1, 26 indicate the project is behind schedule or over budget. These are fairly standard measures of project performance (Archibald, 1976; Kliem et al., 1998; Gido et al., 2000). Participants assessed these measures during regularly scheduled status report meetings. Therefore the time spent in each status report meeting was also evaluated. Similar to design performance, planning time is a measure of project performance. Only projectrelated work was included in this measure. Time associated with completing questionnaires and forms was not included. 2.7.2 Personnel Subsystem Performance Measures Mental workload was the measure reflecting the state of the personnel subsystem. Those who conduct research on workload recommend using multiple measures to assess the effects on humans (Gopher et al., 1986; Wierwille et al., 1993; Veltman et al., 1993; Vincente et al., 1987; ODonnell et al., 1986). The most common dimensions for measuring workload include primary performance, physiological measures, and subjective measures. Research on stress, workload, and fatigue has reported similar consequences for several of the categories. For example, primary performance has been found to degrade in the presence of stress, extreme workloads (mental or physical), or when the human is mentally fatigued. Therefore, the use of primary measures of performance alone would be confounding (Wastell, 1991). Similarly, physiological consequences have been affected by stress, mental fatigue, and/or workload. To isolate the specific causes of the changes in primary performance measures, subjective reports are important. Subjective reports have been found to find differences that primary and physiological measures may not have been sensitive enough to distinguish. 2.7.2.1 Overview of Workload Assessment Initially with physical workload, interest focused on the amount of energy used to accomplish a task (Grise, 1997; Rodahl, 1989). Due to a change in the nature of tasks, from physically intensive to mentally challenging, mental workload has also become an area of interest. If the human is viewed as an information processor, then mental workload can be thought of as the amount of processing capacity that is consumed while an individual performs a task (Eggemeier, 1988). Measuring mental and physical workload is important to assist the understanding of how individuals respond to increases in workload, the potential degradation in performance associated with increased workload, and the general state of an individual. Mental workload has been described as the disjoint between imposed and perceived demands (Jorna, 1992). Others have described workload as a cost that an operator incurs while performing a task (Gopher et al., 1986; Sirevaag, Kramer, Wickens, Reisweber, Strayer, & Grenell, 1993; Rolfe et al., 1973). However, performance does not degrade as a result of mental workload until the amount of effort required to perform the task exceeds the individuals capabilities or limits (Jorna, 1992). In the 1960s and 1970s researchers tended to describe mental workload as a unitary construct (Sirevaag et al., 1993). More recently mental workload is recognized as multi-dimensional and therefore must be measured with multiple metrics (Gopher et al., 1986; Wierwille et al., 1993; Veltman et al., 1993; Vicente et al., 1987; O'Donnell et al., 1986). Multiple measures are particularly critical when the study is conducted in a new environment (Veltman et al., 1993). As mentioned previously, three common categories of workload measures include performance measures (primary and secondary tasks), subjective ratings, and physiological measures. ODonnell and Eggemeier (1986) recommended researchers choose their workload measures based on five selection criteria: sensitivity, diagnosticity, intrusiveness, implementation requirements, and operator acceptance. 27 Sensitivity refers to how well a particular workload measure can differentiate between significant variations in workload that are caused by a task. Diagnosticity is the ability of a workload measure to differentiate between the individuals capacities that are being used in order to complete the task. Intrusiveness refers to the extent to which a measurement technique will negatively impact primary task performance. The ease by which a particular measurement technique can be implemented refers to implementation requirements. The final selection criterion, operator acceptance, refers the how well an individual will agree to actually use a measurement technique. Primary Measures. Primary measures tend to be insensitive to changes in mental workload levels, especially in the low-mid range levels of workload (Meshkati, 1988). High workload situations can be differentiated by primary task measures; whereas low workload levels might not be differentiated due to the individuals ability to increase their efforts to maintain expected performance levels (Meshkati, 1988; Williges et al., 1979). Several researchers agree there is difficulty in generalizing performance measures to other task situations because experimental studies tend to be controlled (Meshkati, 1988; Hicks & Wierwille, 1979). Typical primary measures include accuracy, speed, and number of errors (ODonnell et al., 1986). The primary measures that will be used in this study include design performance, project performance, time, and number of errors made while building the system, which reflect the technological subsystems performance. Subjective Measures. Subjective measures can be collected using direct and indirect inquires of the subjects opinion regarding workload (Meshkati, 1988). One caution with the use of subjective rating scales is the potential for subjects to underestimate their workload level (Svensson et al., 1993). Results of the rating scales might be influenced by the subjects perception of task difficulty (Meshkati, 1988). Moray (1988) stressed the importance of taking individual differences into account when assessing workload. Workload is not merely a property of the task, but of the task, the human, and their interaction (p. 130). (Moray, 1988) The rating scale used to assess mental workload in this study was the NASA Task Load Index (TLX, Hart et al., 1988). Because the physical work associated with design was expected to be very small, physical work was not measured as the scales were unlikely to be sensitive to slight changes in physical work. However, it should be pointed out that physical demand was one of the scales used to determine the NASA TLX. The NASA TLX is a multi-construct rating scale consisting of mental demand, physical demand, temporal demand, performance, effort, and frustration level, where (adapted from Hart et al., 1988): Mental Demand (Low/High): The amount of mental and perceptual activity is required to complete a task, for example thinking, decisions, calculations, recall, and searches. Was the task easy or demanding, simple or complex, exacting or forgiving? Physical Demand (Low/High): The amount of physical activity required to complete a task, for example pushing, pulling, twisting, turning, etc. Was the task easy or demanding, slow or brisk, slack or strenuous, restful or laborious? Temporal Demand (Low/High): The time pressure experienced that was caused by the rate or pace at which the task(s) or task elements occurred. Was the pace slow and leisurely or rapid and frantic? Performance (Good/Poor): The level of success in meeting the goals of the task that were set by either yourself or the experimenter. Effort (Low/High): How hard did you have to work (mentally and physically) to accomplish your level of performance? 28 Frustration Level (Low/High): How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task? After a task is completed, the subject conducts a series of fifteen paired comparisons between each of the constructs to determine which factor in each contributed most to the workload experienced. These comparisons are used to weight ratings in the determination of an overall workload measure. The developers recommended using 12-cm line anchored using bipolar adjectives for the rating scale. The overall workload is found using the following equation: W (t n ) = (wn1v(MD n1 ) + wn 2 v(PD n 2 ) + wn3 v(TD n3 ) + wn 4 v(OPn 4 ) + wn5 v(E n5 ) + wn 6 v(Fn 6 )) / W ni In this equation, the workload estimate for the nth individual is represented by t, W equals the weight, and v is the raw score for each of the respective subscales. Weights are found from the pair-wise comparisons between each factor. Each time a factor is selected as the most significant workload source, the weighting of that factor is increased by one. The overall workload is then found by summing the rating for each factor, multiplied by its weighting and divided by the sum of all the weights. The advantage of task specific weights is that the two sources of variability in ratings that have been identified within tasks (subjects workload definitions) and between tasks (task-related differences in workloads drivers) would be represented from the perspective of the raters (p. 170). (Hart et al., 1988) This point has been frequently questioned in the literature (Nygren, 1991). Several studies have shown that there was essentially no significant difference in the overall measure of workload based on the weighted average compared to an average that considered each construct to have an equal weighting (Nygren, 1991). A validation study for the NASA TLX was conducted to learn if the sub-scales characterized workload across different tasks, if the weightings for the factors distinguished sources of workload and if the weighting procedure provided a global measure sensitive to changes within and between tasks (Hart et al., 1988). To validate the NASA TLX, six male subjects completed thirteen experimental tasks. Across similar tasks, task-related factors were similar. The factors were regressed against the overall workload rating. The factors explained a range of 0.78 to 0.90 of the variance. Weights were found to distinguish between sources of loads. The method of collecting data from the NASA TLX was also explored. The comparisons found strong agreement between computer and verbal (correlation = 0.96), computer and paper/pencil (correlation = 0.94), and verbal and paper/pencil (correlation = 0.95). For test-retest reliability, the relationships were similar (correlation = 0.83). Physiological Measures. Changes in physiological measures of workload are used to estimate the workload level (Meshkati, 1988; Wastell, 1991; ODonnell et al., 1986; Roscoe, 1993; Kramer, 1991; Wierwille & Connor, 1983). Similar to stress, both physical and mental workload affect physiological measures (Rolfe et al., 1973). Changes in activation and arousal impact several physiological responses (Rolfe et al., 1973). Examples of physiological measures include, but are not limited to, respiration, brain potentials, blink rates, pupil diameter, heart rate (HR), and heart rate variability (HRV). In the present study, participants were allowed to freely move around the design area. For participants with automated planning and tracking tools, they needed to move between the work table and the computer workstation. The simple act of moving can impact HR and HRV. A large body of research has shown posture changes, caffeine, nicotine, alcohol, and movement influence cardiovascular activity (Fried et al., 1984). In a field study comparing the use of several CAD systems, heart rates were affected by movement in 70% of the trials (Springer, Langner, Luczak, & Beitz, 1991). In another study, postures significantly affected HR (Lane, Phillips-Bute, & Pieper, 1998). Due to the potential number of confounding factors and the tendency for the apparatus to distract the participants, physiological measures were not included in this study. 29 2.7.2.2 Workload at the Group Level Limited research has been conducted on studying the effect of mental workload on teams and groups (Bowers et al., 1997). One study that measured workload, stress, and fatigue considered twoperson teams of pilots (Hart, Houser, & Lester, 1984). A correlation was found between overall workload (r = 0.65), stress (r = 0.63), and effort (r = 0.66). From this research, the investigators suggested the amount of communication might be suitable as a workload indicator. The majority of the empirical research involving teams and stress or workload manipulated workload as an independent variable (Bowers et al., 1997). Most studies workload analyses were at the individual level (Hart et al., 1984; Thorton, Braun, Bowers, & Morgan, 1992). The studies that aggregated the individual workloads typically used an average after verifying the within team variance was not significant (Beith, 1987). Studies on the effect of workload on team performance conducted in the 1960s found communication degraded performance (Kidd, 1961; Johnston et al., 1968; Johnston, 1966; Naylor et al., 1965; Williges et al., 1966; Bowers et al., 1997). To distinguish which workload measurement techniques were sensitive to changes in workload caused by communication, Casali and Wierwille (1983) systematically varied communication (frequency and combination of call signs) to measure workload using sixteen different measurement techniques including primary, secondary, physiological, and rating scales. Seven workload measures were able to distinguish between different levels of workload caused by communication including the Modified Cooper-Harper Scale (MCH) and Multi-Descriptor Scale (subjective measures); time estimation (secondary task measure); pupil diameter (physiological measure); errors of omission, errors of commission, and communications response time (primary task measures). One of the important outcomes from these studies was that communication requirements increased workloads (Casali et al., 1983; Bowers et al., 1997). Later research on communication further supported that communication adds to the workload. In one study using two-person teams of pilots, there was a significant correlation between communication (per minute of flight) and overall workload (r = 0.65), stress (r = 0.63), and effort (r = 0.66) (Hart et al., 1984). This research led Hart (1984) to suggest that communication might be suitable as a workload indicator for groups (Hart et al., 1984). Teams under high workloads spoke more words; however, low workload teams had more taskoriented communication (Urban, Bowers, Morgan, Braun, & Klein, 1992). In a later study, the type of workload was isolated to time-pressure workload and resource workload (Urban, Weaver, Bowers, & Rhodenizer, 1996). Teams under high resource workloads had a reduction in their communication while teams under time-pressure workloads did not have a reduction in communication. An important finding was that high time-pressure workloads resulted in performance degradations, while high resource workloads did not affect performance. The development of a subjective team workload scale was a component of a study conducted by Beith (1987). In this study, individuals and dyads worked on locating targets on a grid. Half of the dyads were allowed to communicate freely. A wall through which audio communication was allowed separated the other half of the dyads. Beith found that the workload perceptions were lower when individuals worked in teams; however performance in teams did not improve compared to individual performance. He also found that the NASA TLX, a measure of overall workload, and the measure of team workload were correlated. As with studies of individuals, teams made trade-offs between timeliness and accuracy with the associated cost of making more errors to accelerate processing (Serfaty et al., 1990; Johannsen, Levis, & Stassen, 1994). Serfatys (1990) hypothesis was team coordination strategies evolve from explicit coordination under low workload conditions to implicit coordination as workload increase (p. 394). Explicit coordination referred to overt communication and implicit coordination referred to the anticipation of team member actions and requirements (Serfaty et al., 1990). The best performance and workload trade-offs occurred when individuals had partially overlapping responsibilities. Another finding 30 was under high workload, team coordination tended to break down. The focus of effort tended to be on task work, not teamwork. When the task load was unevenly distributed amongst team members, the team functioned as a group of individuals. These findings supported the team evolution and maturation model (Glickman et al., 1987; Salas, Morgan, & Glickman, 1987), which was used by Salas (1992) to develop the integrated team performance model. Specifically, performance was found to be function of both task skills and team skills. Subsequent studies on coordination have confirmed that high performing teams adapted their coordination styles under stress from explicit when workload was low to implicit when workload was high (Serfaty, Entin, & Volpe, 1993; Urban, Bowers, Monday, & Morgan, 1993). To maintain alignment of mental models of functioning, Serfaty, Entin and Volpe (1993) stressed that explicit coordination is critical under low workload levels. Thus when workloads become high, the team members better anticipated the needs of other members. Providing groups with decision aids has been found to increase mental workload when the intent behind the decision aid was to reduce workload (Johannsen et al., 1994). When the group (or an individual) had the option of whether or not to use the aid, that decision process alone added to the workload level. In a review of the literature, Dyer (1984) found several common themes summarizing the impact of workload on teams. Increases in workload affected both individual responsibilities and the individual's responsibility to the group. In addition, holding all factors constant except for team workload, performance degraded as the workload increased. 2.7.3 Job Satisfaction The measurement of job satisfaction has received significant attention in the literature. There tends to be two schools of thought on how to measure satisfaction. One approach is the global feeling of satisfaction about a job and the other is a faceted approach to satisfaction (Spector, 1997). The faceted approach is used to determine if specific parts of a job produce satisfaction. In the faceted approach to measuring satisfaction, since a job is a set of relationships between tasks, roles, responsibilities, interaction, incentives, and rewards, then in order to understand satisfaction, all of the constituent elements of the job must be understood (Lock, 1976). There are many components to job satisfaction including pay, promotion, verbal recognition, working conditions and agents (Locke, 1976). These components are tied directly to Maslows (1943) need hierarchy, Vrooms (1982) expectancy theory, and Herzbergs (1966) motivation-hygiene theory. Of the components, the ones relevant to this study include working conditions and agents. Working conditions are usually taken for granted. However, the extent to which the surroundings are not dangerous or uncomfortable might influence an individuals satisfaction. This effect will depend to some extent on the individuals perceived value of the physical surrounding and how these surrounding help or hinder achieving the work goal (Locke, 1976). In terms of the self, satisfaction can be influenced by the amount of self-esteem, defensiveness, the ability to receive criticism, and the degree to which challenging tasks are valued and pleasure is gained through achievements (Locke, 1976). Co-workers will influence satisfaction to the degree that they help an individual to achieve a common goal, have similar characteristics, and are personal (for example, have discussions that are not job-related). There is little consensus on how to measure job satisfaction nor is there consensus on elements to include in job satisfaction measurements. Many researchers support a multi-faceted approach to measuring job satisfaction (Hackman et al., 1975). This approach is most useful when the investigator wants to determine if certain aspects of a specific job cause satisfaction (Spector, 1997). A global approach is desired when the individuals overall attitude towards the job is of interest. Global and faceted measures have consistently been high and positively correlated (for example, Quinn & Sheppard, 1974). 31 In this study, Quinn and Sheppards (1974) challenge, comfort, resource adequacy, and when applicable, relations with co-workers satisfaction factors will be used. These factors are from the 1972-73 Quality of Employment Survey. Several of the factors and questions were not relevant to this study and were omitted, for example promotion and pay. In addition, the scale was changed to a 7-point Likert-type scale to facilitate the data analysis because the intent of using this questionnaire was not for comparing to a specific population. The original survey was administered to a sample of 1496 individuals living within the United States or the District of Columbia. The internal reliabilities for the constructs in the 1973 application of the survey included facet-specific job satisfaction (0.92), comfort (0.69), challenge (0.83), and resource adequacy (0.87). 2.7.4 Group Process The measurement of group process has a history of being unreliable, complex, as well as lacking sensitivity and measuring irrelevant variables (Dyer, 1984). Interaction analysis, the technique most often used to capture group process, enables the systematic observation of individuals interacting with other individuals and individuals interacting with objects in the environment (Jordan and Henderson, 1995). Trained observers can analyze the interaction either real-time or post-hoc using audio-video taped recordings (Bales, 1950; Williges et al., 1966; Morgan et al., 1986). Generally interaction analysis has seven difference areas of interest (Jordan et al., 1995). The first focus is the structure of events. Investigators are concerned with the chronology of events, when an event starts and stops, which event precedes and follows an event, and how the transitions are made. The second focus is on the temporal organization activity. This refers to the ordering of speaking and nonverbal activity. Rhythms and repetitions in activity are noted. Turn taking is the third focus. Investigators are interested in studying the process of a conversation. This includes who speak and when they speak, the transition between speakers, the bodys position when the speaker speaks, and the turns with artifacts. The fourth focus refers to participation structures. This includes the extent to which participants have a common attentional focus. Studying eye contact, tones of voice helps to understanding how coordination and collaboration occurs within a group. Trouble and repair is the fifth focus. This has also been referred to as communication breakdown (Kies, 1997). Trouble occurs when unspoken rules are violated. Repairs are how these violations are compensated. The sixth focus is the spatial organization of activity. Basically, the main interest is how people use space to express behavior, for example displaying a power relationship and making an apology. The final focus is on the use of artifacts and documents. Sometimes the artifact is the focus of an interaction, other times, artifacts are peripheral. The lack of consistency in method and approach is a commonly reported problem in group process measurement (Bales 1950; Dyer, 1984; Baker et al., 1992). The content of the group interaction has ranged from simple counts of statements by participants to more sophisticated approaches (Bales, 1950; Williges et al., 1966). Several group researchers use frequency counts and time durations of specific behaviors (Meister, 1985). Variations of interaction analysis have also periodically surfaced. For example, behaviorally anchored scales were used to assess specific behaviors in a group (Shiflett et al., 1985). This variation resulted in low reliabilities between evaluators, but the results were promising because nave judges were used. A literature review of group process measures was conducted by Baker and Salas (1992). Some of the key concerns highlighted include a need to identify and clearly present the dimensions of skills and behaviors to be observed. In addition, the format of the tool used to capture data regarding group process needed to be easy to use. One potential format Baker and Salas identified was a behaviorally based checklist (originally developed by Smith and Kendall, 1963). Checklists have been used with some success in studying tactical decision-making in military teams. The Critical Team Behaviors Form 32 (CTBF) is an example of a checklist (Morgan, et al., 1986; Oser et al., 1989; Glickman et al., 1987). The CTBF, which was developed from critical incidents, helps the observer to document the following behaviors: communication, cooperation, coordination, adaptability, ability to give and receive suggestions and criticisms, and team spirit and morale. Another method that is used is to simply rate each team member on a variety of specific behaviors, for example aggressiveness, and then to make an overall assessment of a broader skill, for example coordination (Stout et al., 1994; Brannick et al., 1993). The CTBF has also been used with the augmentation of assessing the behaviors impact on performance (as cited in Baker et al., 1992; Brannick et al., 1993). Therefore when all aspects of the CTBF are employed, the occurrence and frequency of a behavior is recorded and the impact of that behavior on performance is assessed. The initial checklist used different forms for ineffective and effective behaviors. However, through subsequent testing and use of the instrument, these behaviors were combined into a single form for each major behavioral category (Glickman et al., 1987). Using the CTBF, characteristics that distinguish between effective and ineffective teams have been observed (Morgan et al., 1986; Glickman et al., 1987; Oser et al., 1989). In an initial study, Morgan (1986) identified coordination, communication, and adaptability as key behaviors differentiating between effective and ineffective tactical decision making teams. In a follow up study, Glickman (1987) observed thirteen tactical decision making teams. They observed that effective teams exhibited 66% more effective behaviors than ineffective teams. Several validation studies have been conducted on the rating portion of the CTBF instrument. In a study by Brannick (1991 as cited in Baker et al., 1992), the reliabilities of team members, on-site and off-site observers were compared. The on-site observers had the highest reliability estimates (ranging from 0.57 to 0.81), followed by the off-site observers (ranging from 0.36 to 0.74), and self-ratings (ranging from 0.21 to 0.60). The multritrait-multimethod matrix (MTMM, Campbell & Fiske, 1959) was used to find evidence of validity. The method variables were the three sets of ratings (self, on-site and off-site) and the traits were the teamwork variables that were rated. There was little convergence using this method, with on-site and self-raters indicating the highest convergence. Brannick, Roach, and Salas (1993) collected communication frequencies for twenty-six variables, rating scales judgments were made by self-ratings, on-site and off-site observers for giving suggestions, accepting suggestions, cooperation, coordination, and team spirit. They found that about 80% of the correlations between observers communication frequency variables were above 0.70. For the rating scales, the agreement between observers tended to be high; however team members did not show much agreement. Based on a MTMM, there was evidence that the variables giving suggestions and cooperation had convergent and discriminate validity. Overall, the MTMM failed to meet criteria set by Campbell and Fiske. They found that coordination and giving suggestions were two aspects of the team process that can be evaluated with strong agreement between observers, which supported earlier work (Oser et al., 1989). These observations were correlated with outcome assessments. They found when studying communication, capturing visual information was critical. Their off-site observers listened to audio taped recordings and the convergence between observer ratings was stronger with self-ratings and on-site ratings compared to off-site ratings. The difference in what the observer was looking for during each trial affected the ratings that each type of observer made. Observers either collected frequency data or made subjective evaluations of behaviors. Communication frequencies were slightly correlated with subjective evaluations of team process, but strongly correlated to team outcomes. Therefore, observers attending to skills might overlook some important communication behaviors. A modified version of the CTBF was used to capture group process in this study. 33 Chapter 3 Experimental Design The purpose of this chapter is to describe how this study was conducted. A laboratory experiment was performed to address the research questions. As reported in Chapter 1, the following is a summary of the research questions addressed: 1. How was performance (cost effectiveness and design cycle time) affected by team design and project support during the design projects life-cycle? 2. What were the changes in mental workload due to variations in team design and project support during the projects life-cycle? 3. How was project performance (planning time, cost variance and schedule variance) affected by team design and project support? A 2x3x3 mixed-factor design was used to answer the first two questions. Table 3.1 shows the design matrix used to analyze data collected in each design phase. Table 3.1 Design matrix for experiment1 Engineering Analysis and Design Conceptual Design (DP1) Individuals (TD0) None (Control, PS0) Manual Tools (PS1) Automated Tools (PS2) None (Control, PS0) Manual Tools (PS1) Automated Tools (PS2) n1-6 n7-12 n13-18 n19-24 n25-30 n31-36 Preliminary Design (DP2) n1-6 n7-12 n13-18 n19-24 n25-30 n31-36 Detailed Design (DP3) n1-6 n7-12 n13-18 n19-24 n25-30 n31-36 Groups (TD1) 1 TDi=level of team design, PSi=level of project support, DPi=design phase, and ni=ith participant The independent factors investigated included project planning and tracking support (PS) with three levels including no project support, manual project support tools, and automated project support tools; team design (TD) with two levels including individuals and groups of three; and the engineering design phase (DP), which included conceptual design, preliminary design, and detailed design. Project support, a between subjects factor with three levels, referred to the availability of planning and tracking tools to assist in planning and managing the project, for example, creating a scoping document, work breakdown structure, developing a budget, and devising a schedule via a Gantt chart. The levels of project support included: None (Control, PS0): No project support or training on project management was provided. Participants in this treatment did not have a planning phase nor did they track their performance during the design process. Manual (PS1): The project planning and tracking tools were executed by hand, using pencil and paper. Automated (PS2): Participants used computer software designed specifically for assisting in managing projects. Team design, a between subjects factor with two levels, referred to the method in which participants were organized. The levels of team design included: 34 Individual (TD0): An individual worked independently on the engineering design project. Group (TD1): A group of three participants worked together to complete the design project. The engineering analysis and design process, a within subjects factor, referred to the stages of implementing the engineering design process. Included within implementation were three design phases: Conceptual Design (DP1): In conceptual design, the goal was identified, design criteria were specified and potential design alternatives were generated. Preliminary Design (DP2): The list of concepts was narrowed down to two potential ideas. Highlevel layouts were developed and life-cycle costs were estimated. Tradeoff analyses were conducted comparing the alternative designs. To assist the evaluation process, two prototypes were created; one of which was recommended for detailed design. Detailed Design (DP3): The design was formalized by detailed drawings, documenting production procedures, and material selections. At the end of detailed design, any prototypes will be dismantled. The third research question used two research designs. A 2x2 between subjects factorial design was used to analyze the planning phase. Refer to Table 3.2 for the relationship between the independent variables (team design: individuals and groups and project support: manual project tools and automated project tools). During the planning process, the project was defined and goals were established. The deliverables from planning included a scoping document, a baseline plan for the three phases of engineering design, including a breakdown of the activities that occurred in each design phase with a starting and completion time for each activity in the form of a Gantt chart, allocated resources for each task, and a budget. Only participants with manual or automated project tools planned the project. Table 3.2 Design matrix for the 2x2 factorial between subjects design Individuals (TD0) Groups (TD1) Manual Tools (PS0) n7-12 n25-30 Automated Tools (PS1) n13-18 n31-36 During the implementation of the project, participants with project support provided status reports (RP) at 30 minute intervals. During the status reports, the project performance was evaluated to determine if the schedule was on time and on budget. A 2x2x3 mixed factorial was used to analyze the data from status reports. The relationship between dependent variables is shown in Table 3.3. Table 3.3 Design matrix for the 2x2x3 mixed factor design1 Report Period 1 (RP1) N7-12 n13-18 n25-30 n31-36 Report Period 2 (RP2) n7-12 n13-18 n25-30 n31-36 Report Period 3 (RP3) n7-12 n13-18 n25-30 n31-36 Individuals (TD0) Groups (TD1) 1 Manual Tools (PS0) Automated Tools (PS1) Manual Tools (PS0) Automated Tools (PS1) RP=report period While not a formal part of the design process, a prototype of the final system was built and tested after the completion of detailed design. The system was tested to determine if it met the specifications for reliability, manufacturability, size, and robustness. 35 3.1 Subjects One of the issues in planning this study was the number of subjects required to have meaningful and interpretable results. For within subjects factors, there is concern about learning effects. The within subject variable was the design phase. Each phase was unique and required different activities; therefore, ordering effects were not considered. There were six between subjects treatment combinations. Based on experimental design literature, the standard minimum difference (*) between two extreme treatment effects can be used to determine the appropriate number of repetitions (Hinklemann & Kempthorne, 1994; Neter, Kutner, Nachtsheim, & Wasserman, 1996). Another way to consider * is the difference between the maximum and minimum treatment effect divided by the standard deviation of the error. To simply this process, Bowman and Kastenbaum (1975) developed tables containing various numbers of treatments, replicates, and values of 1- (as reported in Hinklemann et al., 1994). The number of replicates required to detect the standard minimum difference (*) is based on the probability of detecting an existing difference (1-) and the number of treatments. The range of the number of replicates, shown in Figure 3.1, is based on 1- = 0.8 and 1- = 0.7. Six replicates per treatment involving 36 trials and 72 subjects were adequate to detect a difference of two standard deviations based upon the figure. Only by adding many additional subjects would additional replicates make improvements in the sensitivity of the planned experiment. There is a decrease in the cost-benefit tradeoff with the number of subjects much greater than six. 6 5 1-B = .8 1-B = .7 4 3 2 1 0 0 5 10 15 20 25 30 Number of Replicates * Figure 3.1 Relationship between number of replicates and the standard minimum difference (*) Examination of data from previous research also helped to identify the number of replicates. Results from Merediths (1997) study was used to provide a starting point for determining the number of replicates required to detect differences. Group size and the presence (or lack) of computer supported cooperative work were two of the independent factors that were varied that were similar to the factors being explored in this study. For each of the main effects, * was calculated and the corresponding number of replicates required to detect a minimum difference was determined. The result of this analysis for the design performance measure, cost effectiveness, is reported in Table 3.4. The number of replicates were determined using a power of 1- = 0.8 and = 0.05, 0.01. Table 3.4 Number of replicates required for detecting a difference in cost effectiveness Computer Supported Cooperative Work (CSCW) Group size = 0.1 >25 8 = 0.05 >25 10 36 The number of replicates was also calculated for conceptual design time, overall process time, satisfaction, and the number of feasible ideas. When the mean square of the error term was not reported, the overall standard deviation was used. Based on this analysis, more than twenty-five replicates would have been required to detect a difference for the conceptual design time, overall process time and satisfaction. The number of replicates for the number of ideas is reported in Table 3.5. Table 3.5 Number of replicates required for detecting a difference in number of ideas CSCW Group size = 0.1 >25 28 = 0.05 >25 34 A study by Beith (1987) provided data to explore the number of replicates required to determine a difference using the NASA TLX and team workload. Even though the task was a target searching task and not a design task, looking at this study was useful because it provided general guidance for the number of replicates. Group type was manipulated between dyads that freely communicated and dyads that did not freely communicate due to separation by a wall. Based on the analysis summarized in Table 3.6 and Table 3.7, at least twelve replicates were needed to determine a difference in the NASA TLX and six replicates were sufficient to detect a difference in overall team workload. Table 3.6 Number of replicates required for detecting a difference in the NASA TLX Group type = 0.1 12 = 0.05 16 Table 3.7 Number of replicates to detect a difference in overall team workload Group size = 0.1 6 = 0.05 8 Data from a study by Harvey (1997) that explored various communication methods for collaborative engineering design, indicated fourteen replicates were required to detect a difference in workload during the first design phase (conceptual design) as shown in Table 3.8. More than twenty-five replicates were required in the second and third design phases. Communication method was the factor manipulated. The media included face-to-face, audio, or audio-video communication. Table 3.8 Number of replicates required for detecting a difference in overall team workload Media = 0.1 14 = 0.05 17 Based on existing data, the number of replicates required to detect differences was discouraging. However, the factor levels in the current study were expected to have more extreme effects compared to the previously studied treatments. For example, in the literature, performance differences were reported to be more discernable between individuals and groups of two and three than the comparison between groups of three or more participants. Project support tools were varied between the extremes of no support to automated support. The subjects involved in this research were individuals and groups of three undergraduate engineering students. The participants were selected to have the following characteristics: Age: In general, age was not expected to influence performance in this study. 37 Gender: Men and women were encouraged to participate in this study. Participants were selected on a first come-first select basis, provided they could attend both a training session and the trial, until all cells were filled. Education: Participants were sophomores, juniors, or seniors in the College of Engineering. At this level, it was expected that participants had at least one course in engineering design. Specifically, participants were expected to have completed the Introduction to Engineering Design course (or the equivalent), a course in which first year students participant in a handson team design project. Familiarity: It was not desirable for group members to have a previous working knowledge of each other. In projects, it is common for a group of people to join together with limited previous experience. These people remain together for the life of the project and then disband. Familiarity was assessed post-hoc and only two members on one team reported previous experience working together. Participants in the manual and automated project support treatments received project management training. Due to the possibility that over time, the information provided in training would be forgotten, training was provided several times. Participants were required to complete the trial no more than ten days after the training (eight days was the longest elapsed time between training and the trial). The participants were tested to verify they had the appropriate skills. The participants in the control (no project support) did not receive project management training due to the bias that would have been introduced. All participants received design training. 3.2 Materials and Equipment All participants had the following tools available: engineering paper, mechanical pencils, erasers, engineering scales, a 45-45-90 triangle, a 30-60-90 triangle, a calculator, a Technic II set (No. 1032) of LEGOTM parts, and parts from various standard LEGOSTM sets (Nos. 3033, 5935, and 4115) with over 400 pieces, rubber bands, tape, and a 4 x 7 piece of cardboard. The system test site materials consisted of a ping-pong ball resting on a 3/8 LEGOTM part (5/8 x 5/8), a one foot long 2 x 4 board, a 2 long piece of tape, and a table that was at least 3 x 6. In treatments with manual project support, participants had unlined paper, lined paper, a calculator, eraser, mechanical pencil, and several pens and markers. In treatments with automated project support, participants had Microsoft Word, and Microsoft Project. Microsoft Project is a software package that provides support for scheduling, assigning resources, budgeting, and tracking project performance. From survey studies, Microsoft Project was reported to be one of the most common project management tools used by project managers in multiple industries and service organizations (Fox & Spence, 1998, Pollack-Johnson et al., 1998). It was unlikely that the participants had prior experience using the project software. Therefore, participants that were in the automated project support treatments were trained to criteria on how to use the software specifically for creating a Gantt chart, allocating resources and developing a budget. They also received training on how to measure their project performance. Many projects in organizations share common resources. Therefore participants in automated project support treatments had access to a pre-entered resource sheet for this project. 3.2.1 Engineering Design Project The engineering design task was Mission Possible, which required the design of a system to move a payload from a starting point to a finish line (Meredith, 1997). The distance between the starting 38 point and finish line was 3 feet as shown in Figure 3.2. Blocking the path was a wall (a one-foot long, 2 x 4 board). The payload (a ping pong ball) had to move over or around the wall. The energy to move the ball could have been either potential energy, for example a rubber band held in tension, or kinetic energy, for example a LEGOTM motor. Once the process began, humans were not allowed to interact with the system. For a complete description of the task, refer to Appendix A.2. Note, Meredith (1997) provided an in depth discussion of the solution set for this task. Figure 3.2 Diagram of the system test site System robustness was also evaluated if the system satisfied the operational requirement of moving over or around the wall. Robustness referred to the ability of a system to satisfactorily perform under a range of conditions that could occur (Clausing, 1994). For this task robustness was evaluated using a golf ball in place of the ping-pong ball. A time and cost constraint was introduced to make the project more realistic. In the treatments without project support, the participants had two hours to complete the task. They received guidelines for the deliverables, for example conceptual design activities must be completed before starting preliminary design. For the treatments involving project support, the participants were given two hours and fifty-five minutes to complete the task. The additional fifty-five minutes was to allow for planning. The time constraints explicitly did not include the time for the scales and questionnaires administered after each phase. The cost constraint was prorated to account for group size: Individuals = $445 and Groups = $685. The cost constraints were based on Merediths (1997) results. The mission possible task was chosen because it was a novel task that the subject pool was unlikely to have been exposed to previously. This task has been used in a previous study, which lent itself to a better estimation of the number of replicates required to detect minimum differences. Furthermore, the task appeared to be adequate for studying the various features of design that were of interest, for example the generation of design concepts, the modeling of a design with drawings, and the testing of the overall design were components of this task. This task could be subdivided into several interrelated work components, which was one of the characteristics of a project. In addition, this task could be completed in four hours. 3.2.2 Supplemental Forms Prior to beginning the trials, each participant completed a questionnaire to collect background information, which helped to identify some of the populations characteristics, and scheduling information. This form can be found in Appendix A.1. Much of the data that was collected during this experiment required participants to complete rating scales. After each phase was complete, subjects completed the NASA Task Load Index (NASA TLX), which assessed mental workload. If the treatment calls for a group, then the group workload scale was administered. The form was the job satisfaction questionnaire that included supplemental questions regarding design and, when relevant, planning and tracking tools. The forms were administered in this order after each phase. 39 3.2.3 Script A script was read to help minimize the differences between each trial. A copy of this script is included in Appendix A.3. 3.3 Facility The laboratory setting for this study was the BESTEAMS Laboratory in 2401 J.M. Patterson Hall on the University of Maryland, College Park campus. The facility had one computer workstation, a worktable, and videotaping capability. All group sessions were videotaped. 3.4 3.4.1 Procedure Pilot Testing A pilot test was conducted for this research. The purpose of pilot testing was multifold: 1. Script: The script was tested to ensure all directions were clear and no instructions omitted. 2. Task: The engineering design project and the associated materials were tested for clarity and understanding. 3. Data Collection: Ensured the collection instruments had no errors and the directions were clear. 4. Time constraint: Determined if the time constraints to complete planning and design tasks were reasonable. 5. Cost Constraints: Determined if the cost constraint was reasonable. 6. Training Materials: The training materials for project management, LEGOTM, detailed drawing, life-cycle costs, and three-view drawing and LEGOTM assembly were reviewed and tested for clarity, errors and understanding. 7. Software: The computer with Microsoft Office and Microsoft Project software was used to ensure proper functioning with multiple software programs running simultaneously. As a result of the pilot test, the following changes were made: 1. Script: Several sections of the script with directions for planning were re-written to clarify instructions. 2. Task: A list of specific outputs from planning and each design phase were listed as part of the task to help remind participants what was required from each phase of the trial. 3. Data Collection: The verbal instructions for the NASA TLX were clarified to instruct the participants to use a single vertical line to indicate their response on the rating scale. 4. Time Constraints: Initially, only thirty minutes was allowed for the planning phase. However in the first pilot the time was greatly exceeded (over an hour). The main issue was the time required to develop a work breakdown structure was excessive. Therefore, a work breakdown structure (based on the design process) was provided to the participants and the participants were allowed to make changes within several constraints. For example, high level activities had to remain the same (conceptual design, preliminary design, detailed design, and manufacturing and testing). In addition, at least two lower level activities had to be associated with each of the high level activities. Even after making this change, the 30 minute time limit was exceeded, but all groups and individuals did finish within forty-five 40 minutes. Therefore, fifty-five minutes was selected as the time limit to ensure all individuals and groups would be able to complete the planning process. The trials without project support took about three hours to complete, while the trials with project support required a maximum of four hours to complete. Note these times included the time to complete the scales and questionnaire after each phase. 5. Cost Constraint: The cost constraint was reasonable based on the pilot test. 6. Training Materials: The training materials for project management, LEGOTM, detailed drawing, life-cycle costs, and three-view drawing and LEGOTM assembly were reviewed. Typographical errors were identified and corrected. The training for Microsoft Project was changed. The first participant in the pilot was instructed to use the on-line training provided by the software. Training of additional participants was changed to specific instructions on how to create tasks, link tasks, enter time durations, assign resources, make changes, view budgets, and view the resource allocation sheet to identify problems with scheduling. This resulted in a shortening of time required to train participants. 7. Software: The software programs could be open simultaneously. 3.4.2 Experimental Procedure The following steps occurred prior to conducting the research: 1. Approval of the Institutional Review Board (IRB). IRB approval was secured from Virginia Tech and the University of Maryland. The approved documents are provided in Appendix A.5. 2. Participants were solicited using email and visits to classroom, in which professor permission was previously secured. The procedure used to solicit participants is provided in Appendix A.3. Students who agreed to participate in the research were asked to complete a demographic form and schedule. Participants were randomly assigned to each of the treatments. Once the assignment was determined, participants were contacted via email and phone to confirm their scheduled training and laboratory time. 3. Participants completed the IRB consent forms prior to participating in training and agreed not to disclose the nature of the experiment or training until after the last participant completed the task. 4. If participants were assigned to a treatment requiring manual or automated project tools, they completed a project management training session, which was conducted by the investigator. The training was based on that used by the Virginia Tech Leadership Development Institute (Van Aken, 2001). The participant had to demonstrate they could write a scoping document, work breakdown structure, create a Gantt chart, and calculate schedule and cost performance indices. They also had to pass a terminology test with a 70% or better. 5. All participants had to complete training on design, manufacturing and life-cycle cost calculation. The following steps occurred once participants arrived to the testing facility: 6. Participants were given a copy of their signed informed consent forms, which included the investigator contact information. 7. Participants with project support were led through the planning process according the script provided in Appendix A.3. All participants were led through the engineering design process according to the script (refer to Appendix A.3). 41 The remaining procedures are sub-divided based on the phase. After each phase, the data collection instruments were administered in the following order: the NASA TLX, group workload scales (if a group treatment), the job satisfaction questionnaire with supplemental questions related to planning and tracking (when applicable) and design. After the system was built and tested at the conclusion of the design process, participants again completed the forms reflecting back over the entire experience. 3.4.2.1 Planning Only participants in treatments with manual or automated project support participated in planning. Participants without project support were not instructed to plan their project; they immediately began with the conceptual design phase. Planning began after the instructions for planning were read. The start time was entered into the time recorders log (the purchaser in group conditions) just after the instructions were completely read and questions answered. The participants had to plan the tasks involved with designing a system to satisfy the mission possible task, including generating ideas for potential solution, creating and testing two prototypes, selecting and documenting a single solution. While a work breakdown structure was provided, the participants were instructed that they could make changes provided they did not change the highest level activities: conceptual design, preliminary design, detailed design, and manufacturing and testing, and under each main activity, they had to have at least two sub-tasks. The additional constraint was that each main activity had to be completed in sequence; in other words, detailed design could not begin before preliminary design ended. The planning phase ended once the participant(s) notified the researcher that he/she or they had a scoping document, schedule (in the form of a Gantt chart), a work breakdown structure with time duration estimates and resource assignments (in terms of personnel), and a cost estimate (budget). 3.4.2.2 Conceptual Design For participants without project support, the conceptual design began after the conceptual design instructions were read. For participants with project support, conceptual design began once the participants were told they could begin to implement their plan for conceptual design along with several reminders of the required products for conceptual design. The phase ended once the participant(s) notified the researcher that he/she or they had a goal, a set of design criteria, and a list of potential design concepts. 3.4.2.3 Preliminary Design For participants without project support, the preliminary design began after the reading of the preliminary design instructions. For participants with project support, preliminary design began after the participants were told they could begin to implement their plan for preliminary design and were reminded of the deliverables for that phase. During this phase, the design ideas generated during conceptual design were narrowed down to two feasible ideas. Then, high level drawings were created and life-cycle costs estimated. During the tradeoff analysis, the participants had the opportunity to prototype the designs and compare cost estimates. The phase concluded once the participant(s) selected a single system. The products of preliminary design included a tradeoff analysis between the two design concepts documenting support for the selection of the system, a life-cycle cost estimate, high level drawing of the system, and a single concept selected for detailed design. 3.4.2.4 Detailed Design Detailed design began after detailed design instructions were read for participants without project support. Participants with project support began detailed design after being told they could implement 42 their plan for detailed design and were reminded of the deliverables. The start time was recorded after the instructions were read. The phase ended once the participant(s) had detailed drawings, a bill of materials, detailed instructions for manufacturing the system, and a rough life-cycle cost for the system (which could not be completed until after the testing was complete). All participants were told they could not continue to prototype during detailed design. If they needed to check the fit between two or three LEGOSTM pieces, that was permitted. After detail design was complete, the prototypes were dismantled. 3.4.2.5 Manufacturing and Testing Testing of the final system was not considered part of the formal design process. As long as the participants completed the design process within the two hours, not including the time to complete the assessments, they were allowed to build and test their designs. All participants completed the design process; however, anecdotally, some indicated that they would have done things differently had they had more time. The purpose of building and testing was to determine if the final system achieved the requirements. The system was built according to instructions created during detailed design. The system was tested three times. If the system successfully passed the barrier (the wall) regardless of crossing the line at least once, then the system was tested for robustness. To test for robustness, the ping-pong ball was replaced with a golf ball. The system was tested three times with the golf ball. Participants with project support conducted the testing according to their plan, while the participants without project support were led according to the script. During testing, the number of errors made during building the system and the time required for building and testing the system were recorded. 3.5 Dependent Variables The effects of the independent factors were assessed for design performance, planning and tracking performance, human performance, and organizational design effectiveness. Design performance was measured by cost effectiveness (a function of system effectiveness and life-cycle cost; Blanchard et al., 1990). Design performance also included the time in each phase and design cycle time. Planning performance included planning time, status report meeting time, cost performance index, and schedule performance index. The NASA TLX was used to capture individual workload to assess human performance. If participants were in a group treatment, the group workload was measured. Organizational design effectiveness was assessed using a faceted job satisfaction questionnaire. 3.5.1 Design Performance Design performance was defined by cost-effectiveness (Blanchard et al., 1990). Cost effectiveness was a first-order parameter comprised of two second-order parameters: life-cycle cost and system effectiveness. The method for calculating cost effectiveness is provided in Table 3.9. Another measure was the time in design phase and overall design cycle time. Design cycle time was the sum of the process times for the individual design phases (Meredith, 1997). Time associated with completing the scales and questionnaire was not included in process time. 43 Table 3.9 Components of cost effectiveness (adapted from Blanchard et al., 1990; Meredith, 1997) Component Cost Effectiveness A. System Effectiveness 1. Performance a. Range & Accuracy b. Reliability c. Maintainability d. Speed e. Transportability f. Producability g. Size, Weight e. Robustness 2. Operational Availability B. Life-cycle Cost 1. R & D Cost 2. Investment Cost a. Design Cost b. Data Cost c. Test and Evaluation d. Manufacturing Cost e. Inventory Cost 3. Operation and Support a. Maintenance Cost 4. Phase-Out Cost Calculation System Effectiveness (A)/Life-cycle Cost (B) Performance Range & Accuracy + Producability + Size Rate on a scale between 0 500 (described in Appendix A.7) N/A N/A N/A N/A Rate on a scale between 0 500 (described in Appendix A.7) Rate on a scale between 0 500 (described in Appendix A.7) Rate on a scale between 0 500 (described in Appendix A.7) N/A Investment Cost + Operation and Support Cost N/A Design Cost + Manufacturing Cost + Test & Evaluation Cost Conceptual Design Man-hours x ($60/hr) + Preliminary Design Man-hours x ($60/hr) + Detailed Design Man-hours x ($60/hr) N/A Man-hours x $60/hour Man-hours x $60/hour + Material Cost N/A Maintenance Cost $5/moving part + 10% Material Cost (for spares) N/A 3.5.2 Planning and Tracking Performance Planning performance was measured with planning time, status report meeting time, Gantt chart and scoping document scores, cost performance index (CPI), and schedule performance index (SPI). These measures were captured for treatments with project support. Status report meeting time, CPI, and SPI were calculated at each status report meeting. The indices are related to cost variance, the difference between the actual and planned expenses, and schedule variance, the difference between the actual and planned time required to complete the project. To facilitate the interpretation, the variances were converted into a ratio format. Ratios equal to one indicated the performance was going as planned while values greater than one indicated better performance than planned and less than one indicated worse performance. These are typical measures used to assess project performance (Gido et al., 2000). The indices were reported as part of the status report, which occurred every 30 minutes. 3.5.3 Mental Workload The NASA TLX was the rating scale used to capture the subjective perceptions of workload (Hart et al., 1988). The scale is based on a weighted average of six factors: mental demand, physical demand, temporal demand, performance, effort, and frustration. Each factor was explored in addition to the overall weighted average. A validation study conducted for the NASA TLX found it indexed to a global measure of mental workload that was sensitive to changes within and between tasks (Hart et al., 1988). The justification for using this scale was presented in detail in Section 2.7.2.2. 44 When the level of analysis was between groups, additional subjective rating scales were used to capture workload that was specific to group interaction. These scales were adapted from Beiths (1987) team workload scales and captured the amount of work that was required for coordination activities that typically are not required for individual work (Beith, 1987; Bowers et al., 1997). In the work conducted by Beith, a strong correlation was found between the team workload measure and the NASA TLX measure of overall workload. 3.5.4 Job Satisfaction Job satisfaction was assessed based on a faceted job satisfaction measure. The faceted survey was adapted from Quinn and Sheppards (1974) quality of employment survey. The factors that were included are challenge, comfort, and resource adequacy. An overall faceted satisfaction score was the sum of all responses. The internal consistency reliabilities for these factors ranged from 0.66-0.87 (refer to Section 2.7.3 for the complete discussion). 3.5.5 Group Process Group process measures included the amount of effective and ineffective behaviors related to communication, coordination, cooperation, adaptability, team spirit and morale, and the ability to give and receive suggestions and criticisms (termed feedback). A modified version of the Critical Team Behaviors (CTB) Form was used to capture this behavioral data. The form was modified to accommodate non-military observation. Two observers were used to provide a reliability check. In previous studies using the CTB Form, observers tended to have stronger agreement than team members, which can be interpreted that observers were more reliable than team members (Brannick et al., 1993). Observers were trained to criteria on how to use the CTB form using the video tapes from the pilot studies for research. Each behavior was briefly explained and then the observers viewed a video with examples of the various behaviors. Using the CTB form they recorded the behaviors. The training video was reviewed with the observers and deviations were discussed. Pearsons Product Moment was calculated to compare the level of agreement. The raters also completed the group workload scales rating the value of the group interaction, the difficulty of the interaction, the degree of cooperation, and the overall workload. Inter-rater reliabilities were calculated to determine the level of agreement in the ratings. 3.6 Data Analysis The first statistical test conducted on the data was a multivariate analysis of variance (MANOVA). The purpose of the MANOVA was to reduce the possibility of type I error when conducting multiple ANOVAs on dependent measures. For the MANOVA to indicate significance, the mean difference must be larger in the multivariate compared with the univariate analysis of variance (ANOVA). Note that in many cases, the underlying assumptions of MANOVA were not met (normality and homogeneity of variance). However, as a general test the MANOVA was an adequate indicator. The preferred method of analysis for between subjects models was analysis of variance (ANOVA). The two underlying assumptions for ANOVAs are normality and homogeneity of variance. Normality was examined using a normal probability plot of the residuals. If normality was not obvious from the plot, the Shapiro-Wilks Lambda test was used to test for normality. The null hypothesis for the Shapiro-Wilks tests was that the distribution is normal. If normality was satisfied then homogeneity of variance was explored visually. If the data were suspicious then Hartleys test was conducted. If both assumptions were satisfied, ANOVA tests were conducted. 45 In the between subject models, if at least one assumption was violated, a variation of ANOVA was attempted. The variation was to systematically group the variance (first ungrouped, which was the standard ANOVA method to create a baseline for model comparison, then grouping by each factor) and to compare the variance models based on a goodness of fit value to the baseline. Several fits could have been selected, but the one used, the corrected form of Akaikes Information Criterion (AICC), was robust to situations in which there was no variance (Akaike, 1973; McQuarrie & Tsai 1998). A lower number indicated a better fit. The variance grouping with the lowest AICC value was selected (which removes the violation of homogeneity of variance) and the normality was rechecked based on the grouping. In systematically comparing the AICC, the value had to be at least 2 points lower than the AICC for the baseline. This is a rule of thumb developed by statisticians to indicate a worth while improvement between models. Then the variance analysis was calculated using only the relevant portion of the variance in the error terms from the variance grouping. One deviation from the standard ANOVA reporting style was that the degrees of freedom were not used in the calculation of the F-value. The degrees of freedom were estimated only to determine the probability level. Note that the degrees of freedom for the error term reported in significant F-values often did not match the degrees of freedom reported in the variance analysis table for the residuals. The standard estimate for degrees of freedom reported in the table was Z2*2 (Littell, Milliken, Stroup & Wolfinger, 1996). This variance analysis method was preferred to a data transformation because the interpretation is similar to standard ANOVA. If this method did not resolve the violation, the data were transformed. Two different transformations were made depending on the data (Howell, 1992): a. If the variance of each factor level was proportional to the mean of the factor level, a square root transformation was used. The transformation is Y ' = Y or Y ' = Y + Y + 1 , where Y is the transformed observation and Y is the original observation. b. If the standard deviation of the error term for each factor level was proportional to the mean of each factor level, a logarithmic transformation was used. The appropriate transformation is Y = log Y or Y = log10(Y+1) if Y was between 0 and 1. Depending on the skewness of the data, the log transformation may have negatively influenced the normality. In these cases the data were reflected using: maximum(Y+1) - 1, and then making the logarithmic transformation. For data that could be transformed, satisfied the ANOVA assumptions, or could be analyzed using the variance grouping, a decision level of 0.05 was used to determine statistical significance. The decision level indicated the probably of the null hypothesis occurring less than five times in one hundred will result in rejecting the null hypothesis (Martin, 1996). In reporting the significance, the actual p-value was used. The standard ANOVA tables were reported using the common format (refer to Table 4.9). The first column showed the source of the variation. The second column had the degrees of freedom, DF, for each source. The third column was for the sum of squares, SS. The fourth column was for the mean sum of squares, MS; the fifth column was for the F statistic, and the last column contained the probability level. A graphical system was used to indicate significance (* indicated p<0.05, ** indicated p<0.01, and *** indicated p<0.001). The variance grouping procedure was the standard method used to analyze the variance in variables with mixed models. A different table was used to report the analysis of variance with variance grouping (refer to Table 4.11). The first column indicated if the effect was fixed or random, the second column provided the degrees of freedom, DF, for each source, the third column indicated the variance component for the random effects, the forth column contained the statistic from the F-test and the fifth column contained the probability value. Note that variance components were not computed for the fixed effects when the variance grouping was used. This method is referred to as PROC MIX in SAS. 46 Post hoc analyses were conducted if there were significant main effects and/or interactions. Least significant differences (LSD) were used to perform pairwise multiple comparisons between the levels of the significant effect. This is the least conservative of all of the pair comparison tests because there is no correction for an inflated alpha. A decision level of p<0.05 was used to determine significance. Because individuals were compared with groups, the level of analysis was important to consider prior to collecting data. Project and design performance were directly comparable between the individuals and groups. The method for comparing the workload of groups with individuals was not well documented in the literature. A large number of team-level workload studies manipulated workload as an independent factor and only measured primary task performance (Kidd, 1961; Johnston et al., 1968; Urban et al., 1992; Urban et al., 1994; Urban et al., 1996). Other investigators used an individual level of analysis for subjective measures of mental workload or did not report the method used to obtain the team measure (Thorton et al., 1992; Hart et al., 1984; Whitaker et al., 1997). However, team level measures can be found by exploring other group literature domains. One common method explored was a comparison of individual scores to the best/worst score in the group (Smith, 1989). However, subjective measures (mental workload and job satisfaction) were aggregated for team members in order for a comparison with individuals to be possible. The method of creating a team score was to average the individual scores and check for significant variation between the members (Zaccaro & Lowe, 1988; Kleiner & Drury, 1987; Beith, 1987). Large variations in the measures indicated that members were affected differently, which provided insight on the group process. In addition, correlations should exist between the averaged scores within the team and the team measure. 3.7 Assumptions and Limitations The purpose of this section was to refine the scope of this research by providing assumptions and limitations. Assumptions are statements that did not have research to justify them. The limitations identified what was not covered by this study. Assumptions: The participants in this study had participated in at least one engineering design project. The participants in this study were familiar with reading and creating hand drawn engineering drawings. During the freshman year, all engineering students took a course in engineering design during which they participated in an engineering team design project. Aspects of the design project included developing a solution to a problem, creating a graphical representation of the problem, and building a physical model of the design. To help ensure a minimum level of ability subjects were trained to criterion in reading a three-view drawing. In addition, subjects were trained to criterion for hand drawing a detailed engineering drawing. Most students did not have previous experience with the project management life-cycle and project tools. But students could be trained to criteria in a reasonable amount of time (about 1-2 hours) on how to create a work breakdown structure, a Gantt chart, allocate resources and develop a budget. In addition, students could be quickly trained to criteria on how to track their performance. Participants would retain their training over the course of time that elapsed between training and the trials. To help minimize this issue, the maximum time that elapsed was eight days (where the trial occurred on the eighth day; however this amount of time occurred only once). The groups were novice teams. 47 Limitations The use of hand drawing during detail design was not reflective of a true design environment. For the purpose of this initial study, the actual design process, not the tools, was believed to be the more important factor. Contextual variables related to the subjects gender and age were not considered, even though the data was collected. The purpose of this study was not intended to test specific computer hardware and software products. The products were chosen due to availability, accessibility, and/or representation within industry. Micro-ergonomic issues associated with the software and manual tools were not considered. The Mission Possible task was selected for the novelty of the task and the ability to complete the task using the engineering design process. The limits are that this task does not reflect a true design situation because each trial is limited to four hours. Thus the external validity was questionable, but the task did enable a starting point from which further research can be investigated. The use of students raised another issue with external validity. Most students in engineering are not proficient designers mainly due to a lack of experience. Because students were readily available and possessed the basic skills for engineering design, they were used in this study. The threat to validity is noted but here again, they provided a cost effective method for a starting point. 48 Chapter 4 4.1 Results for First Order Variables Participant Demographics The participants were 72 sophomores, juniors and seniors enrolled in the A. James Clark School of Engineering at the University of Maryland. The first 72 students who indicated interest and could be scheduled for training and the trial were selected. Participants were randomly assigned to treatments. Figures 4.1-4.6 contain demographic information on the participants. In these figures the treatment groups are abbreviated: GA = groups with automated project support, GM = groups with manual project support, GN = groups without project support, IA = individuals with automated project support, IM = individuals with manual project support, and IN = individuals without project support. Figure 4.1 contains the range of ages of participants in each treatment group. The outliers in the figure are denoted with an asterisk (*). The outliers were determined to be data points that were outside the upper (Q1+1.5(Q3-Q1), where Q1 is the first quartile and Q3 is the third quartile) and lower limits (Q1-1.5(Q3-Q1)). Figure 4.2 contains the average grade point average for the participants. 35 Age 30 25 20 GA GM GN IA IM IN Treatment Figure 4.1 Box and whisker plot of age range for treatment groups (GA=group automated, GM=group manual, GN=group none, IA=individual automated, IM=individual manual, IN=individual none, and *=outlier) 49 4 GPA 3 2 GA GM IA GN Treatment IM IN Figure 4.2 Grade point average range for each treatment Gender composition is provided in Table 4.1. Overall twenty-nine percent of the participants were female and seventy-one percent were male. Over fifty percent were from mechanical engineering, followed by twenty-two percent from aerospace engineering, and ten percent from electrical engineering. The other majors represented were biological resources, civil engineering, chemical engineering, computer engineering, fire protection engineering, and nuclear engineering. Table 4.1 Gender composition in treatments Treatment % Women Group automated 22% Group manual 33% Group none 17% Individual automated 67% Individual manual 50% Individual none 17% Figure 4.3 provides the academic level of the participants. Twenty-five percent were sophomores, forty-three percent were juniors, and thirty-two percent were seniors. 50 70% 60% 50% 40% Soph Jr Sr 20% 10% 0% GA GM GN IA IM IN Treatment Percent 30% Figure 4.3 Academic level of the participants Participants self reported the number of engineering design projects they had worked on (shown in Figure 4.4). On average, participants in the individual treatments without project support reported having the most experience with design projects (mean=7.00, sd=11.33), followed by groups with manual or automated support (manual group: mean=3.61, sd=4.29; automated group: mean=3.61, sd=3.31), groups without support (mean=2.94, sd=1.662), individuals with manual support (mean=2.83, sd=1.17) and individuals with automated support (mean=2.5, sd=1.23). Everyone worked on at least one project. 30 20 No. Projects 10 0 GA GM GN IA Treatment IM IN Figure 4.4 Box and whisker plot of self-reported participation in engineering design projects (* indicates an outlier) Each participant also indicated whether or not they were familiar with project management (PM). The number of participants responding yes (indicated by 1, with a maximum value of three for groups) or 51 no (indicated by 0) is provided in Figure 4.5. One group with automated support was unfamiliar with project management prior to the study and one individual with automated support and two with manual support were not familiar with project management. All participants that were provided with project support tools had to pass a short test on project management terminology and demonstrate that they could develop a scoping document, work breakdown structure, a Gantt chart, a budget, and use the cost performance index and schedule performance index formulas before they could participate in the study. 3 Familiarity with Project Management 2 1 0 GA GM GN IA IM IN Treatment Figure 4.5 Box and whisker plot of self reported familiarity with project management (PM) 4.2 Descriptive Statistics The mean cost effectiveness and standard deviation for in each treatment groups is shown in Table 4.2. Table 4.2 Mean cost effectiveness for each treatment Team Design Group Individual Project Support Automated Manual None Automated Manual None Mean 1.45 1.22 1.32 1.21 2.15 1.67 Standard Deviation 0.70 0.55 0.82 0.75 2.81 0.72 The mean life-cycle cost and standard deviation for in each treatment groups is shown in Table 4.3. The constraint for groups was $685 and for individuals was $445. Life-cycle cost consisted of material cost and design cost. The mean material cost and standard deviations for each treatment group are shown in Table 4.4. Design costs in each treatment group are reported in Table 4.5 52 Table 4.3 Mean life-cycle cost for each treatment Team Design Group Individual Project Support Automated Manual None Automated Manual None Mean $671.72 $645.24 $675.34 $537.88 $554.89 $488.92 Standard Deviation $187.08 $191.06 $224.22 $167.92 $193.89 $129.320 Table 4.4 Mean material costs and standard deviations for each condition Team Design Group Individual Project Support Automated Manual None Automated Manual None Mean $342 $272 $279 $383 $381 $320 Standard Deviation $160 $149 $183 $143 $159 $110 Table 4.5 Mean design cost in each treatment group Team Design Group Individual Project Support Automated Manual None Automated Manual None Mean $250.17 $280.00 $310.50 $85.67 $93.83 $104.50 Standard Deviation $46.70 $35.53 $46.19 $18.40 $12.61 $14.36 The mean system effectiveness and standard deviation in each treatment groups is shown in Table 4.6. The mean reliability and standard deviations for each treatment group are shown in Table 4.7. Table 4.6 Mean system effectiveness in each treatment group Team Design Group Individual Project Support Automated Manual None Automated Manual None Mean 1007 700 767 567 747 780 Standard Deviation 509 192 269 198 372 270 Table 4.7 Mean reliability and standard deviations for each treatment group Team Design Group Individual Project Support Automated Manual None Automated Manual None Mean 278 356 294 67 206 200 Standard Deviation 122 195 183 133 232 101 53 4.3 Exploring ANOVA Assumptions Prior to conducting the analysis of variance, the data for each variable were explored to determine if the assumptions of normality and homogeneity of variance were met. The first step was to create normal-probability distribution charts for the residuals for each data set. If these charts appeared linear, the data were assumed to be normal. If the data did not appear linear, the Shapiro-Wilkes test for normality was conducted. If the test statistic, W, was below 0.9, the data were assumed to be non-normal. Scatter plots were created comparing the residual by each treatment main effect to determine homogeneity of variance. If the data appeared to have less than a two-fold spread between each of the levels, the variances were assumed to be equal. If the spread was more than two-fold, a statistical analysis was conducted to determine if the variances were equal. An overview of the analysis on dependent variables is shown in Table 4.8. (The order of the variable presentation in the two summary tables does not reflect the order in which the variables are presented in this chapter). The first column provides the name of the variable. The second column indicates the analysis of variance method used. For all mixed designs, the PROC MIX procedure in SAS (SAS Institute Inc, Cary, North Carolina) was employed. The method for this procedure is to check the variance grouping by levels in a factor and determining the best fit to the mixed design model. The tests of fit are reported in Appendix A.9. The term Var grouping in column two indicates the SAS proc mixed analysis of variance method was used (DP refers to variance grouping by design phase; TD refers to grouping by team design; and PS refers to grouping by project support). Other terms reported in column two include ANOVA, indicating that the standard analysis of variance method was used; or Transform indicating that a data transformation was conducted due to a violation in the assumptions and then an ANOVA was conducted (the transformation method is also reported in column two). Column three contains the significant effects from the analysis and includes the F value and associated probability. SAS was used to analyze all mixed designs. If the normality assumption was violated, SPSSTM was used to transform the data and conduct ANOVAs for the transformed data. For between subject designs, MinitabTM and SPSSTM were used to test the assumptions of normality and homogeneity of variance, SPSSTM was used to run the standard ANOVA tests and multiple comparison tests. In a few between design cases that violated the homogeneity of variance assumption, the data were analyzed in SAS using the PROC MIX procedure. The reason is the analysis from PROC MIX was simpler to interpret compared an ANOVA analysis on transformed data. The fourth column contains significant differences found from the multiple comparison tests (where CD=conceptual design, PD=preliminary design, and DD=detailed design). 54 Table 4.8 Summary of data analysis1 Dependent Variable Planning Analysis Method Significant Effects Significant Comparisons Time NASA TLX Job satisfaction Design Process ANOVA ANOVA ANOVA Var grouping: DP TD: F(1,20)=8.492** PS: F(1,20)=9.849** Group>Individual Automated<Manual Time DP: F(2,30)=173.15*** DP*TD: F(2,30)=4.40* NASA TLX Var grouping: TD DP: F(2,38)=26.63*** DP*PS: F(4,38)=4.12** Job satisfaction Status Reports Var grouping: TD Var grouping: U Var grouping: Report Period Var grouping: Report Period ANOVA Log10(x+1) ANOVA ANOVA Var grouping: TD Var grouping: TD DP: F(2,39)=4.46* TD: F(1,20)=4.58* Period: F(2,40)=4.77* Period: F(2,18)=11.71*** Period: F(2,12)=12.34** CD<PD=DD DD: Group< Individual Individual: CD<PD=DD Group: CD<DD<PD CD<PD<DD CD: Automated=Manual>None PD: Automated>None Automated: CD<DD Manual: CD<PD=DD None: CD<PD<DD CD=PD>DD Group>Individual Period1=Period2>Period3 Period1>Period2=Period3 Period1>Period2=Period3 Status report time CPI SPI Overall (Reflective) Cost effectiveness Design time Number of errors NASA TLX Job satisfaction PS: F(2,30)=3.586* Automated<None *p<0.05 **p<0.01 ***p<0.001 1 In the table, TD= team design, PS=project support, DP=design phase (CD=conceptual design, PD=preliminary design, DD=detailed design), and U=ungrouped 4.4 4.4.1 Performance Performance consisted of measures associated with design and project planning and tracking. Design Performance Cost Effectiveness The main variable for consideration in design performance was cost effectiveness. Recall that cost effectiveness was computed based on system effectiveness (numeratorcomprised of reliability, robustness, producibility, and size) and life-cycle cost (denominatorcomprised of labor cost, material 55 cost, and maintenance costs). No significant differences were found for cost effectiveness as reported in Appendix A.10. Design Cycle Time Design cycle time was measured as the time required for the subject(s) to complete each design phase. The time started after the instructions for the design phase had been reviewed and questions answered. The time ended when the subject(s) told the investigator all of the design tasks in that phase were complete. While no time limit was set for individual design phases, the participants were informed they had exactly two working hours (not including the breaks to complete questionnaires) to complete the design project. Analysis of variance was used to determine statistical significance as shown in Table 4.9 for the analysis of the total design cycle time. There was a significant difference in the total design cycle time for the project support, F(2,30)=3.586, p=0.04. The Least Significant Difference (LSD) multiple comparisons were conducted to determine which levels differed significantly (Table 4.10). The treatments with automated support tools had significantly lower cycle times compared to treatments without project support. Table 4.9 ANOVA for design cycle time Source DF SS MS F P Team Design (TD) Project Support (PS) TD*PS S/TD*PS Total *p<0.05 1 2 2 30 35 30.25 1392.25 120.50 5823.83 7366.73 30.25 696.08 60.25 194.13 0.156 3.586 0.310 0.696 0.040* 0.736 Table 4.10 Comparisons of mean design cycle time for type of project support mean Automated Manual *p<0.05 Manual 93.6 min 4.42 None 104.0 min 14.83* 10.42 89.2 min 93.6 min Significant effects were found in the analysis of time spent in each design phase. As shown in Table 4.11, design phase was the significant main effect, F(2,30)=173.15, p<0.0001, and the interaction between design phase and team design was significant, F(2,30)=4.40, p=0.0176. From the comparisons in Table 4.12, conceptual design was significantly shorter than the other design phases. 56 Table 4.11 Variance analysis conducted for time in each design phase1 Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within Design Phase (DP) DP*TD DP*PS DP*TD*PS Residual CD Residual PD Residual DD Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 1 2 2 61 2 2 4 4 30 30 30 0.09 2.19 0.19 0 173.15 4.40 1.19 1.05 18.2167 104.1300 196.1500 0.7590 0.1211 0.8281 <0.0001*** 0.0176* 0.3268 0.3896 *p<0.05 ***p<0.001 1 For random effects the variance components are reported, while for fixed effects the F ratios and their probabilities are reported. DF are ordinary least squares degrees of freedom. CD refers to conceptual design, PD refers to preliminary design and DD refers to detailed design. Table 4.12 Comparisons of the mean time in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 42.47 min 30.11** Detailed Design 40.75 min 28.39** 1.72 12.36 min 42.47 min To explore the significant interaction between design phase and team design, LSD multiple comparisons were conducted as shown in Table 4.13. Refer to Figure 4.6 for a graphical representation of the comparisons. In the figure, alphabetic characters are used to indicate significant differences (e.g., treatments labeled with A are similar, treatments labeled with B are similar, but those labeled with A are different from those labeled with B). The mean time in phase was shorter during conceptual design than in the other phases for individuals and groups. Individuals took significantly longer during detailed design than groups. Groups took significantly longer during preliminary design than in detailed design. Table 4.13 Comparisons of mean design time based on design phase and team design mean CD Individual CD Group PD Individual PD Group DD Individual *p<0.05 **p<0.01 CD Group 14.11 min 3.5 PD Individual 39.72 min 29.11** 25.61** PD Group 45.22 min 34.61** 31.11** 5.50 DD Individual 44.33 min 33.72** 30.22** 4.61 0.89 DD Group 37.17 min 26.56** 23.06** 2.56 8.06* 7.17* 10.61 min 14.11 min 39.72 min 45.22 min 44.33 min 57 50 40 30 20 A 10 0 CD A B B B C Mean Time (min.) Individual Group PD Design Phase DD Figure 4.6 Average time spent in each design phase based on team design (Alphabetic characters represent differences between treatments. Treatments labeled A are similar and those labeled B are similar. Treatments labeled A are different from those labeled B.) 4.4.2 Planning Performance Project Planning Time For the treatments with project support, the time to plan the project (included creating a scoping document, work breakdown structure, and Gantt chart) was measured. Time started after the instructions for project planning had been reviewed and questions answered. Time ended when the participant(s) told the investigator that all of the planning tasks were complete. Participants were informed they had 55 minutes (not including the break to complete questionnaires) to complete the planning activities. Analysis of variance was used to determine statistical significance as shown in Table 4.14 for the analysis of the planning time. There were two significant main effects: team design, F(1,20)=8.492, p=0.009, and project support, F(1,20)=9.849, p=0.005. Groups (mean=48.83, sd=8.88) took significantly more time to plan their design project than individuals (mean=38.00, sd=13.05). Participants with automated project support (mean=37.58, sd=12.80) took significantly less time to complete the planning process compared to participants with manual project support (mean=49.25, sd=8.68). Table 4.14 ANOVA for planning time Source TD PS TD*PS S/TD*PS Total ***p<0.001 DF 1 1 1 20 23 SS 704.167 816.667 266.667 1658.333 3445.833 MS 704.167 816.667 266.667 82.917 F 8.492 9.849 3.216 P 0.009*** 0.005*** 0.088 Status Report Meeting Time Status report meetings were required every thirty minutes. The length of each meeting was recorded. Team design and report period were the significant effects for the average time elapsed during status report meetings as shown in Table 4.15 (team design: F(1,20)=4.59, p=0.0448; report period: 58 F(2,40)=4.77, p=0.0139). Groups (mean = 2.64, sd=1.839) had significantly longer status report meetings than individuals (mean = 1.64, sd=1.313). From post hoc analysis in Table 4.16, the last meeting was significantly shorter than the first two meetings. Table 4.15 Variance analysis for status report meeting times Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within Period Period*TD Period*PS Period*TD*PS Period*S/(TD*PS) *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 1 2 2 20 2 2 4 4 40 4.58 0.00 0.69 0.4313 4.77 2.88 0.10 2.30 0.3621 0.0448* 1.0000 0.4150 0.0139* 0.0677 0.9024 0.1135 Table 4.16 Multiple comparisons of mean status meeting time based on report period mean Period 1 Period 2 *p<0.05 **p<0.01 Period 2 2.58 min 0.2500 Period 3 1.50 min 0.8333* 1.0833** 2.33 min 2.58 min Performance Indices for Planning As indicated in Table 4.17, report period was the significant main effect in the analysis of the cost performance index (CPI), F(2,18)=11.71, p<0.0005. From the post hoc analysis shown in Table 4.18, the CPI at the first status report meeting was significantly larger than the CPI at the second and third status report meeting. Table 4.17 Variance analysis for the cost performance index (CPI) Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within Period Period*TD Period*PS Period*TD*PS Residual Period 1 Residual Period 2 Residual Period 3 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 1 2 2 16 2 2 4 4 18 18 18 4.24 0.000 1.29 0.04184 11.71 0.83 0.86 0.16 0.03665 0.001271 0.02328 0.0563 0.9961 0.2731 <0.0005*** 0.4501 0.4411 0.8530 59 Table 4.18 Multiple comparisons of the mean CPI based on report period mean Period 1 Period 2 **p<0.01 Period 2 1.016 0.1893** Period 3 0.978 0.2274** 0.0381 1.205 1.016 Reporting period was the significant main effect in the analysis of the schedule performance index (SPI), F(1,20)=4.368, p=0.0011, as shown in Table 4.19. From the comparisons in Table 4.20, the SPI reported at the first status report meeting was significantly larger than the SPI reported in subsequent meetings. Table 4.19 Variance analysis for the schedule performance index (SPI) Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within Period Period*TD Period*PS Period*TD*PS Residual Period 1 Residual Period 2 Residual Period 3 **p<0.01 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 1 2 2 11 2 2 4 4 12 12 12 3.85 0.02 0.86 0.03610 12.34 1.00 0.78 0.14 0.03262 0.004046 0.02252 0.0749 0.8881 0.3720 0.0011** 0.3969 0.4796 0.8697 Table 4.20 Multiple comparisons of the mean SPI for each reporting period mean Period 1 Period 2 **p<0.01 Period 2 1.023 0.1854** Period 3 0.974 0.2348** 0.0493 1.209 1.023 4.5 NASA TLX Planning No significant differences were found in the analysis of the NASA TLX during planning. The ANOVA summary table is reported in Appendix A.10. Design Process In the variance analysis conducted on the NASA TLX, the significant main effect was the design phase, F(2,60)=26.64, p<0.0001, and the significant interaction was between design phase and project support, F(4,60)=4.11, p=0.0073, as shown in Table 4.21. From the post hoc analysis (Table 4.22), the mean NASA TLX rating was significantly different in each design phase. The comparisons for the interaction are reported in Table 4.23 and shown graphically in Figure 4.7. Within conceptual design, the 60 NASA TLX was significantly lower in treatments without project support compared to treatments with project support. During preliminary design, the ratings were significantly higher in treatments with automated support compared to treatments without support. For treatments with automated support, the rating was higher during detailed design compared to conceptual design. For treatments with manual support, the rating was significantly lower during conceptual design compared to preliminary design. For treatments without project support, the mean ratings were significantly different in each design phase. Table 4.21 Analysis of the NASA TLX during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual G Residual I **p<0.01 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random 1 2 2 26 2 2 4 4 38 38 0.01 2.68 1.91 3.5208 26.63 1.20 4.12 0.62 0.6815 2.3978 0.9114 0.0873 0.1689 <0.0001*** 0.3130 0.0072** 0.6504 Table 4.22 Multiple comparisons of the mean NASA TLX in each design phase mean Conceptual Design Preliminary Design *p<0.05 **p<0.01 Preliminary Design 14.42 1.3412** Detailed Design 15.18 2.1086** 0.7674* 13.08 14.42 Table 4.23 Comparisons of NASA TLX for the interaction between design phase and project support mean CD Automated CD Manual CD None PD Automated PD Manual PD None DD Automated DD Manual *p<0.05 **p<0.01 14.43 13.57 11.23 15.26 14.61 13.38 15.55 14.99 CD Manual 13.57 0.8691 CD None 11.23 3.2068** 2.3377* PD Automated 15.26 0.8228 1.6918 4.0296** PD Manual 14.61 0.1785 1.0476* 3.3853** 0.6442 PD None 13.38 1.0536 0.1845 2.1532* 1.8764* 1.2321 DD Automated 15.55 1.1202* 1.9893* 4.3270** 0.2974 0.9417 2.1738* DD Manual 14.99 0.5596 1.4287** 3.7664** 0.2632 0.3811 1.6132 0.5606 DD None 15.00 0.5700 1.4391 3.7768** 0.2528 0.3915 1.6236* 0.5502 0.0104 61 20 16 12 8 4 0 Automated Manual Design Phase None AC AC E E D CE CD F E B CD PD DD AE DF AD Mean Rating Figure 4.7 The mean TLX ratings for the interaction between design phase and project support Reflective The analysis of variance conducted on the NASA TLX evaluated after the design project was complete did not reveal significant effects or interactions. Refer to Appendix A.10 for the ANOVA summary table. Number of Errors The number of errors was another measure of workload and was captured during the building of the system. No significant differences were found in the analysis of variance conducted on the number of errors as reported in Appendix A.10. 4.6 Job Satisfaction Planning No significant differences were found in the variance analysis conducted on job satisfaction during planning. Refer to Appendix A.10 for the ANOVA summary table. Design Process Table 4.24 contains the summary of the variance analysis for job satisfaction during design. The significant main effect was design phase, F(2,60)=4.46, p=0.013. From the comparisons in Table 4.25, job satisfaction was significantly lower in detailed design compared to that in conceptual and preliminary design. 62 Table 4.24 Variance analysis for job satisfaction during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual G1 Residual I Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random 1 2 2 27 2 2 4 4 39 39 0.25 0.14 2.01 34.1939 4.46 0.90 0.42 0.52 8.4082 32.6828 0.6210 0.8714 0.1529 0.0180* 0.4157 0.7904 0.7208 *p<0.05 1 In the residual term, G refers to groups and I refers to individuals Table 4.25 Comparisons of the mean job satisfaction in each design phase mean Conceptual Design Preliminary Design *p<0.05 **p<0.01 Preliminary Design 67.39 0.482 Detailed Design 64.90 2.97** 2.49* 67.87 67.39 Reflective In the variance analysis for the reflective job satisfaction, there were no significant differences as reported in Appendix A.10. 63 Chapter 5 5.1 Introduction Results for Second Order Variables Chapter 4 contained the results for the main dependent variables of interest. Several of the primary dependent variables were calculated using intermediate variables. Several supplemental questions were also asked in an attempt to learn more about the impact of independent factors on the design products and the humans involved in the design process. These variables will be presented in this chapter. In this chapter the following abbreviations were used: TD=team design; G=group and I=individual; DP=design phase: CD=conceptual design, PD=preliminary design, and DD=detailed design; PS=project support: A=automated, M=manual, and N=none; U=ungrouped. 5.2 Exploring ANOVA Assumptions Table 4.8 provides an overview of the analysis on dependent variables. The first column provides the name of the variable. The second column indicates the variance analysis method used: ANOVA indicates the standard analysis of variance method; Var grouping indicates the modified mixed analysis of variance method, or Transform: indicates a data transformation was performed and the method of transformation follows the colon. For variables analyzed using between subjects designs, MinitabTM and SPSSTM were used to test the assumptions of normality and homogeneity of variance and SPSSTM was used to conduct standard ANOVA tests and multiple comparison tests. SAS was used to run the variance grouping procedure on all variables analyzed using the mixed model. Appendix A.9 contains the summary for the goodness of fit tests for variables analyzed as a mixed model. And if a transformation was employed, SPSSTM was used to transform the data and run the ANOVA tests. Column three contains the significant effects from the analysis and includes the F value and associated level of significance, the decision level was p<0.05. Column four contains the significant post hoc comparisons. Table 5.1 Summary of the data analysis for supplemental performance variables Dependent Variable System effectiveness Reliability Robustness System size Manufacturing time Life-cycle cost Design cost Analysis Method ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA Var grouping: TD Significant Effects Significant Comparisons TD: F(1,30)=7.383* G>I TD: F(1,30)=4.952* TD: F(1,19)=296.02*** PS: F(1,19)=4.49* TD: F(1,30)=7.747** G>I G>I A<N G<I Material cost Total parts Moving parts Unique parts Concepts Design criteria Gantt chart score Scoping document score *p<0.05 **p<0.01 ***p<0.001 ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA PS: F(1,20)=27.17*** A>M 64 Table 5.2 Summary of the data analysis for supplemental variables Dependent Variable Planning Analysis Method Significant Effects Significant Comparisons NASA TLX Mental demand Physical demand Temporal demand Performance Effort Frustration Job Satisfaction Comfort Excessive work Physical surroundings Perceived time Personal problems Challenge Develop ability Interesting problem Freedom Problem difficulty See results Resources Equipment Information Responsibility Supplemental Questions Best Doubt Ease of use Efficiency Effectiveness Productivity Satisfaction Design Process ANOVA ANOVA: Log10(x+1) ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA Var grouping: TD ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA Var grouping: TD ANOVA ANOVA Transform: Log10(reflected x+1) ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA TD: F(1,20)=5.354* I>G TD*PS: F(1,20)=7.70* M: I>G G: A>M NASA TLX Mental demand Physical demand Temporal demand Performance Performance (outlier) Effort Var grouping: TD Var grouping: U Var grouping: U Var grouping: TD Var grouping: PS Var grouping: TD DP*TD: F(2,32)=6.71** DP: F(2,60)=43.09*** DP: F(2,60)=28.27*** DP: F(2,46)=8.52*** DP*PS: F(4,46)=2.82* DD: G<I G: CD=PD>DD CD<PD=DD CD<PD<DD CD<PD<DD CD: A>N PD: A>M M: CD=PD<DD N: CD<PD=DD CD<PD=DD Frustration *p<0.05 **p<0.01 ***p<0.001 Var grouping: U DP: F(2,60)=13.06*** 65 Table 5.2 Summary of the data analysis for supplemental variables (continued) Dependent Variable Design Process Analysis Method Significant Effects Significant Comparisons Job Satisfaction Comfort Excessive work Var grouping: TD Var grouping: DP DP: F(2,60)=18.26*** DP: F(2,33)=9.73*** DP*TD*PS: F(4,37)=2.96* TD*PS: F(2,30)=4.30* DP: F(2,42)=25.60*** TD*PS: F(2,29)=5.28* DP: F(2,38)=6.78** DP: F(2,37)=7.79** DP: F(2,36)=3.48* DP: F(2,60)=8.26*** DP*PS*TD:F(4,42)=3.30* Physical surroundings Perceived time Personal problems Challenge Develop ability Interesting problem Freedom Problem difficulty See results Resources Equipment Var grouping: U Var grouping: TD Var grouping: PS Var grouping: DP Var grouping: TD Var grouping: TD Var grouping: DP Var grouping: DP Var grouping: U Var grouping: TD Var grouping: DP CD>PD>DD CD>PD>DD I,M: CD=PD>DD I,N: CD=PD>DD G,M: CD>PD G: M<N CD>PD>DD G: A<N N: G>I CD=PD>DD CD>PD=DD CD>PD=DD CD<DD, CD<PD CD,N: I>G I,N: CD>PD=DD Information Var grouping: TD Responsibility Var grouping: TD Supplemental Design Questions Doubt Var grouping: TD Supplemental Project Support Questions Ease of use Var grouping: U Efficiency Var grouping: U Effectiveness Var grouping: U Productivity Var grouping: TD Satisfaction Var grouping: DP Reflective DP: F(2,43)=3.91* DP: F(2,40)=5.40** CD>DD CD>PD=DD NASA TLX Mental demand Physical demand Temporal demand Performance Effort Frustration Job Satisfaction Comfort Excessive work Physical surroundings Perceived time Personal problems *p<0.05 **p<0.01 ***p<0.001 Var grouping: PS ANOVA ANOVA ANOVA Var grouping: TD ANOVA ANOVA Var grouping: TD ANOVA ANOVA Var grouping: TD TD: F(1,30)=4.380* G<I 66 Table 5.2 Summary of the data analysis for supplemental variables (continued) Dependent Variable Reflective Analysis Method Significant Effects Significant Comparisons Challenge Develop ability ANOVA ANOVA TD*PS: F(2,30)=4.76* N: G>I I: A>N Interesting problem Var grouping: PS Freedom Var grouping: TD Problem difficulty Var grouping: TD See results ANOVA Resources ANOVA Equipment Var grouping: TD Information Var grouping: TD Responsibility Var grouping: TD Supplemental Design Questions Best of ability ANOVA Liked system ANOVA Met objectives ANOVA Supplemental Planning Questions Productivity ANOVA Satisfaction ANOVA Remain on schedule ANOVA Remain on budget ANOVA *p<0.05 5.3 Performance Performance consisted of measures associated with design and project planning and tracking. Several supplemental variables were collected to provide insight into the performance in each category. 5.3.1 Design Performance The main variable for consideration was cost effectiveness, which was presented in Chapter 4. To understand the results, the components of the equation used to calculate cost effectiveness were evaluated. Recall cost effectives was a function of system effectiveness (reliability, robustness, system size, producibility) and life-cycle cost (design cost, material cost, and maintenance cost). A MANOVA was conducted on the components in the numerator as shown in Table 5.3. Refer to Appendix A.11 for the complete MANOVA table. Table 5.3 MANOVA to test the affect of reliability, robustness, system size, and producibility on main effects and interactions Source Team Design Project Support Team Design * Project Support *p<0.05 F 4.111 0.697 1.065 P 0.010* 0.692 0.401 The reliability score was the only component from the numerator of cost effectiveness with a significant effect. As shown in Table 5.4, team design was the significant effect, F(1,30)=7.383, p=0.011. Systems designed by groups had higher reliability scores (mean=309.26, sd=163.22) compared to those 67 designed by individuals (mean=157.59, sd=168.55). The variance analysis tables for variables that did not have significant effects are shown in Appendix A.10. Table 5.4 ANOVA for reliability Source TD PS TD*PS S/TD*PS Total *p<0.05 DF 1 2 2 30 35 SS 207025.200 747204.234 20439.006 841231.100 1142899.540 MS 207025.200 37102.117 10219.503 28041.035 F 7.383 1.323 0.364 P 0.011* 0.281 0.698 The denominator of the cost effectiveness equation was life-cycle cost, which was based on material, design, and maintenance costs, number of moving parts (used in calculating the maintenance costs) and indirectly the total parts (impacting material costs) and number of unique parts. A MANOVA was conducted on the components measured in dollars in the denominator as shown in Table 5.5. Refer to Appendix A.5 for the complete MANOVA table. Table 5.5 MANOVA to test the affect of life-cycle, design, and material costs on main effects and interactions Source Team Design Project Support Team Design * Project Support **p<0.01 ***p<0.001 F 121.76 3.546 0.733 P <0.0001*** 0.005** 0.625 There was a significant difference between team designs in the analysis on life-cycle cost, F(1,30)=4.952, p=0.034, as shown in Table 5.6. The mean life-cycle cost for groups (mean=664.10, std=189.754) was significantly higher than that for individuals (mean=527.23, std=158.421). Table 5.6 ANOVA for life-cycle cost Source TD PS TD*PS S/TD*PS Total *p<0.05 DF 1 2 2 30 35 SS 168593.7 3432.047 13884.400 1021452 1207363 MS 168593.7 1716.023 6942.200 34048.412 F 4.952 0.050 0.204 P 0.034* 0.951 0.817 In the analysis of design cost, shown in Table 5.7, team design and project support were significant (team design: F(1,19)=296.02, p<0.0001; project support: F(2,19)=4.49, p=0.0255). The mean design cost for groups (mean=280.22, sd=47.80) was significantly higher than that for individuals (mean=94.67, sd=16.43). As reported in Table 5.8, the design cost for treatments with automated support were significantly lower than the design costs for treatments without project support. There were no significant differences in the analysis conducted on material cost as shown in Appendix A.10. 68 Table 5.7 Variance analysis for design cost Source Effect DF Variance Component F value Probability TD PS TD*PS Residual G Residual I *p<0.05 ***p<0.001 Fixed Fixed Fixed Random Random 1 2 2 19 19 296.02 4.49 1.23 1858.96 234.64 <0.0001*** 0.0255* 0.3135 Table 5.8 Comparisons of mean design cost based on type of project support mean Automated Manual **p<0.01 Manual 186.92 19.00 None 207.50 39.58** 20.58 167.92 186.92 A MANOVA was conducted on the total parts, moving parts and unique parts as shown in Table 5.9. The complete MANOVA is reported in Appendix A.11. Table 5.9 MANOVA to test the affect of total, moving and unique parts on main effects and interactions Source Team Design Project Support Team Design * Project Support F 2.574 1.073 1.287 P 0.074 0.390 0.279 The only measure with a significant difference was the number of moving parts. As reported in Table 5.10, team design was significant, F(1,30)=7.747, p=0.009. Based on the pos hoc analysis, groups (mean=2.22, std=0.88) had significantly fewer moving parts than the individuals (mean=3.28, std=1.41). Refer to Appendix A.10 for the ANOVA tables for total parts and unique parts. Table 5.10 ANOVA for number of moving parts Source TD PS TD*PS S/TD*PS Total **p<0.01 DF 1 2 2 30 35 SS 10.028 5.167 2.722 38.833 56.750 MS 10.028 2.583 1.361 1.294 F 7.747 1.996 1.052 P 0.009** 0.154 0.362 A MANOVA was conducted on the number of concept ideas and number of design criteria (Table 5.11). The full MANOVA table is shown in Appendix A.11. No significant effects were found for number of concepts generated and number of design criteria as reported in Appendix A.10. Table 5.11 MANOVA to test the affect of concepts and design criteria on main effects and interactions Source Team Design Project Support Team Design * Project Support F 2.092 0.189 0.510 P 0.142 0.943 0.729 69 5.3.2 Planning Performance The planning documents were evaluated to determine if there was a difference in the scoping document and the Gantt chart (completeness, readability, and errors) based on team design and project support. A point system was developed to provide consistency between scoring and the highest possible score was five. A MANOVA was conducted on the scoping document and Gantt score as shown in Table 5.12. The full MANOVA table is provided in Appendix A.11. Table 5.12 MANOVA to test the affect of planning documents on main effects and interactions Source Team Design Project Support Team Design * Project Support ***p<0.001 F 0.000 14.078 0.984 P 1.000 <0.0001*** 0.392 There were no significant differences in scoping documents as reported in Appendix A.10. Project support was significant in the analysis on the Gantt chart score, F(1,20)=27.17, p<0.0001 (Table 5.13). The charts developed using automated project support (mean=4.92, sd=.29) had significantly higher scores than those developed using manual project support (mean=2.92, sd=1.24). Table 5.13 ANOVA for the Gantt chart score Source TD PS TD*PS S/TD*PS Total ***p<0.001 DF 1 1 1 20 23 SS 0.000 24.000 0.167 17.667 41.833 MS 0.000 24.000 0.167 0.883 F 0.00 27.17 0.19 P 1.000 <0.0001*** 0.669 5.4 NASA TLX Planning The NASA TLX was a weighted average of six rating scales: mental demand, physical demand, temporal demand, performance, effort, and frustration. A MANOVA was conducted for the ratings scales from planning, as summarized in Table 5.14. Refer to Appendix A.11 for the complete MANOVA results. No significant differences were found in the analysis on the TLX components. Refer to Appendix A.10 for the ANOVA tables. Table 5.14 MANOVA for NASA TLX components from planning Source Team Design Project Support Team Design *Project Support F 2.285 2.316 1.778 P 0.091 0.088 0.171 Design Process To help understand which aspects of the NASA TLX the participants felt demand from during design, the scales were analyzed individually after a multivariate analysis of variance was conducted (Table 5.15). Refer to Appendix A.11 for the complete table. 70 Table 5.15 MANOVA for NASA TLX components during design Source Team Design Project Support Design Phase Team Design * Project Support Team Design * Design Phase Project Support * Design Phase Team Design *Project Support * Design Phase *p<0.05 ***p<0.001 F 0.48 0.51 12.35 0.56 2.07 1.49 1.05 P 0.1390 0.8972 <0.0001*** 0.8593 0.0276* 0.0819 0.4152 As shown in Table 5.16, the interaction between design phase and team design was significant in the analysis of mental demand, F(2,32)=6.71, p=0.0037. From the multiple comparisons (Table 5.17), in detailed design, the mental demand was significantly lower for groups than for individuals. For groups, mental demand was significantly lower during conceptual design compared to detailed design. These differences are shown in Figure 5.1. Table 5.16 Analysis of mental demand during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual G Residual I **p<0.01 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random 1 2 2 23 2 2 4 4 32 32 0.02 1.86 2.34 4.0768 0.55 6.71 0.98 0.43 2.9442 9.1931 0.8847 0.1783 0.1193 0.5815 0.0037** 0.4347 0.7846 Table 5.17 Comparisons of mental demand for the interaction between design phase and team design mean CD Individual CD Group PD Individual PD Group DD Individual *p<0.05 **p<0.01 CD Group 15.10 1.912 PD Individual 14.61 1.423 0.489 PD Group 14.67 1.480 0.432 0.057 DD Individual 15.25 2.061 0.149 0.638 0.582 DD Group 12.92 0.268 2.182** 1.693 1.750** 2.331* 13.19 15.10 14.61 14.67 15.25 71 20 16 Mean Rating 12 8 4 0 CD PD Design Phase DD A AB A AB B Group Individual A Figure 5.1 Comparison of mental demand for the interaction between team design and design phase Design phase was significant in the analysis on physical demand, F(2,60)=43.09, p<0.0001, as shown in Table 5.18. From the post hoc comparisons in Table 5.19, physical demand was significantly lower during conceptual design compared to preliminary and detailed design. Table 5.18 Variance analysis on physical demand during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS s/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 1 2 2 30 2 2 4 4 60 0.09 0.05 0.55 11.0838 43.09 0.39 1.91 1.20 1.5628 0.7641 0.9502 0.5833 <0.0001*** 0.6768 0.1200 0.3216 Table 5.19 Multiple comparisons of physical demand in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 7.12 5.1023** Detailed Design 7.91 5.8991** 0.7968 2.02 7.12 In the analysis on temporal demand, design phase was significant, F(2,60)=20.61, p<0.0001 (Table 5.20). From the comparisons reported in Table 5.21, temporal demand was significantly different in each design phase. 72 Table 5.20 Variance analysis for temporal demand during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS DP*S/TD*PS ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 1 2 2 30 2 2 4 4 60 0.05 0.57 0.79 6.0751 28.27 1.96 2.21 2.03 7.6788 0.8291 0.5703 0.4630 <0.0001*** 0.1500 0.0790 0.1019 Table 5.21 Multiple comparisons of temporal demand in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 13.67 2.6608** Detailed Design 15.91 4.9058** 2.2450** 11.01 13.67 Main effects and interactions in the analysis of the performance rating were not significant. Refer to Appendix A.10 for the summary variance analysis table. Design phase and the interaction between design phase and project support were significant in the analysis on the effort rating (design phase: F(2,46)=8.51, p=0.0007; design phase*project support: F(4,46)=2.82, p=0.0357) as shown in Table 5.22. From the post hoc analysis in Table 5.23, effort was significantly different in each design phase. The multiple comparisons for the interaction are reported in Table 5.24. During conceptual design, effort was significantly higher in treatments with automated project support compared to treatments without project support (Figure 5.2). During preliminary design, effort was significantly higher in treatments with automated project support compared to treatments with manual project support. In treatments without project support, effort was significantly lower during conceptual design compared to preliminary and detailed design. In treatments with manual support, effort was higher during detailed design compared to conceptual and preliminary design. 73 Table 5.22 Variance analysis on effort during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual G Residual I *p<0.05 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random 1 2 2 28 2 2 4 4 46 46 0.04 1.87 2.76 7.2510 8.52 0.13 2.82 0.61 2.8345 6.3969 0.8425 0.1734 0.0806 0.0007*** 0.8809 0.0358* 0.6606 Table 5.23 Multiple comparisons of effort in each design phase mean Conceptual Design Preliminary Design *p<0.05 **p<0.01 Preliminary Design 13.27 1.0388* Detailed Design 14.33 2.0898** 1.0510* 12.24 13.27 Table 5.24 Comparisons of effort for the interaction between design phase and project support1 mean CDA CDM CDN PDA PDM PDN DDA DDM 14.37 11.87 10.47 14.96 12.08 12.79 14.56 13.92 CDM 11.87 2.4951 CDN 10.47 3.8986** 1.4035 PDA 14.96 0.5897 3.0848* 4.4883** PDM 12.08 2.2880 0.2071 1.6106 2.8777* PDN 12.79 1.5789 0.9162 2.3197* 2.1686 0.7091 DDA 14.56 0.1974 2.6925 4.0960** 0.3923 2.4854 1.7763 DDM 13.92 0.4483 2.0468* 3.4503* 1.0380 1.8397* 1.1306 0.6457 DDN 14.49 0.1267 2.6218 4.0253** 0.4630 2.4147 1.7057 0.0707 0.5750 *p<0.05 **p<0.01 1 The abbreviations in the table are as follows: CDA=conceptual design automated support; CDM= conceptual design manual support; CDN= conceptual design unsupported; PDA=preliminary design automated support; PDM= preliminary design manual support; PDN= preliminary design unsupported; DDA=detailed design automated support; DDM= detailed design manual support; DDN= detailed design unsupported 74 16 AC C AC AB AB C B AC AC 12 Mean 8 Rating 4 Automated Manual None 0 CD PD Design Phase DD Figure 5.2 Comparing effort ratings for the interaction between design phase and project support Design phase was significant in the analysis on the frustration rating (provided in Table 5.25), F(2,60)=13.062, p<0.0001. From the post hoc analysis in Table 5.26, frustration was significantly lower during conceptual design compared to preliminary and detailed design. Table 5.25 Variance analysis for frustration during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS DP*S/TD*PS ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 1 2 2 30 2 2 4 4 60 1.97 0.90 0.07 16.0208 13.06 1.25 0.84 0.43 11.4258 0.1706 0.4189 0.9363 <0.0001*** 0.2951 0.5039 0.7886 Table 5.26 Multiple comparisons of frustration in each design phase Mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 10.09 2.7420** Detailed Design 11.33 3.9782** 1.2362 7.35 10.09 Reflective A MANOVA was conducted for the NASA TLX rating scales evaluated after the design project was completed, as summarized in Table 5.27. Refer to Appendix A.11 for the complete MANOVA results. 75 Table 5.27 MANOVA to test the affect of reflective NASA TLX components Source Team Design Project Support Team Design *Project Support F 1.761 0.487 0.693 P 0.148 0.918 0.750 After conducting an analysis of variance on the components that comprised the NASA TLX measure, the only item with a significant effect was frustration: team design was the significant effect, F(1,30)=4.380, p=0.045 (Table 5.28). From post hoc comparisons, groups (mean=9.61, sd=4.20) experienced significantly less frustration than individuals (mean=13.30, sd=5.88) when reflecting back over the entire process. Refer to Appendix A.10 for the ANOVA tables for the ratings that were not significant. Table 5.28 ANOVA for reflective frustration Source TD PS TD*PS S/TD*PS Total *p<0.05 DF 1 2 2 30 35 SS 123.024 7.216 36.992 842.612 1009.844 MS 123.024 3.608 18.496 28.087 F 4.380 0.128 0.659 P 0.045* 0.880 0.525 5.5 Job Satisfaction Planning Job satisfaction was based on answers to a series of questions relating to comfort, challenge and availability of resources. A MANOVA was conducted on comfort, challenge, and resources to determine the affect on main effects and interactions during planning. The results are summarized in Table 5.29. Refer to Appendix A.11 for the complete MANOVA table. During planning, there were no significant differences in the analysis of variance conducted on each variable as reported in Appendix A.10. Table 5.29 MANOVA to test the affect of job satisfaction components on main effects and interactions during planning Source Team Design Project Support Team Design * Project Support F 0.721 1.270 0.119 P 0.553 0.315 0.948 A MANOVA was also conducted on all questions used to evaluate job satisfaction during planning as reported in Table 5.30. Refer to Appendix A.11 for the complete MANOVA results. Table 5.30 MANOVA for responses to questions used to determine job satisfaction during planning Source Team Design Project Support Team Design *Project Support F 0.882 0.854 0.451 P 0.590 0.610 0.900 76 Comfort Comfort was determined based on perceptions of excessive work, physical surroundings, time, and the ability to forget personal problems. The analysis of variance on perceptions of the participants excess work, physical surroundings, and personal problems were not significantly different for the different treatments. Refer to Appendix A.10 for the ANOVA summary tables. In the analysis of the participants perception of having enough time to adequately plan their project (Table 5.31), team design was significant, F(1,20)=5.354, p=0.031, as shown in Table 5.31. Individuals (mean=6.25, sd=0.754) had significantly higher ratings than groups (mean=5.25, sd=1.41). Table 5.31 ANOVA for the perception of enough time for planning Source TD PS TD*PS S/TD*PS Total *p<0.05 DF 1 1 1 20 23 SS 5.999 4.741 0.908 22.407 34.055 MS 5.999 4.741 0.908 1.120 F 5.354 4.232 0.810 P 0.031* 0.053 0.379 Challenge There were no significant effects from the analysis on responses to developing ability, level of interest in the problem, freedom in approaching the job, level of problem difficulty, and ability to see the results of work. Refer to Appendix A.10 for the variance summary tables. Resources The analysis of variance on perceptions having access to right equipment, information, and responsibility had no significant differences. Refer to Appendix A.10 for the ANOVA summary tables. Design Process A MANOVA was conducted to determine the affect of comfort, challenge and resources on main effects and interactions (Table 5.32). The complete MANOVA analysis is given in Appendix A.11. Table 5.32 MANOVA to test the affect of comfort, challenge, and resources during design Source Team Design Project Support Design Phase Team Design * Project Support Team Design * Design Phase Project Support * Design Phase Team Design *Project Support * Design Phase ***p<0.001 F 0.16 0.65 7.36 0.70 0.73 0.85 0.71 P 0.9238 0.6894 <0.0001*** 0.6489 0.6266 0.6034 0.7396 A MANOVA was also conducted on all of the responses used to determine comfort, challenge, and resources to determine the affect on main effects and interactions as shown in Table 5.33. The complete MANOVA analysis is given in Appendix A.11. 77 Table 5.33 MANOVA on responses to questions used to determine job satisfaction during design Source Team Design Project Support Design Phase Team Design * Project Support Team Design * Design Phase Project Support * Design Phase Team Design *Project Support * Design Phase *p<0.05 ***p<0.001 F 2.69 1.27 4.99 0.95 1.43 0.88 1.04 P 0.026* 0.2675 <0.0001*** 0.5473 0.1336 0.6941 0.4273 Comfort Design phase was significant in the analysis on comfort (Table 5.34), F(2,60)=18.26, p<0.0001. From the post hoc analysis in Table 5.35, comfort was significantly different in each design phase. Table 5.34 ANOVA of comfort during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual G Residual I ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random 1 2 2 28 2 2 4 4 45 45 0.36 0.38 1.50 6.3506 18.26 1.22 1.04 0.66 1.5032 4.1070 0.5509 0.6860 0.2404 <0.0001*** 0.3056 0.3988 0.6226 Table 5.35 Multiple comparisons of comfort in each design phase Mean Conceptual Design Preliminary Design *p<0.05 **p<0.01 Preliminary Design 22.20 0.954** Detailed Design 20.79 2.370** 1.417* 23.16 22.20 The questions that were used to determine comfort included perceptions regarding personal problems, physical surroundings, time, and workload. The analysis did not detect differences due to the treatments for personal problems. The interaction between team design and project support was significant in the analysis of physical surroundings (Table 5.36), F(2,30)=4.30, p=0.0228. Based on the multiple comparisons in Table 5.37, the physical surroundings were rated significantly lower in groups with manual support compared with groups without support, as shown in Figure 5.3. 78 Table 5.36 Analysis of physical surroundings during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS DP*S/TD*PS *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 1 2 2 30 2 2 4 4 60 0.53 0.71 4.30 0.4916 0.35 0.52 0.91 0.58 0.1510 0.4713 0.4994 0.0228* 0.7032 0.5985 0.4663 0.6790 Table 5.37 Comparisons of physical surroundings based on team design and project support mean Group automated Group manual Group none Individual automated Individual manual *p<0.05 Group manual 5.11 0.4074 Group none 6.22 0.7037 1.1111* Individual automated 6.11 0.5926 1.0000* 0.1111 Individual manual 5.89 0.3704 0.7778 0.3333 0.2222 Individual none 5.34 0.1296 0.2778 0.8333 0.7222 0.5000 5.52 5.11 6.22 6.11 5.89 7 6 Mean Score 5 4 3 2 1 AB A B AB A AB Group Individual Automated Manual Project Support None Figure 5.3 Interaction between project support and team design for physical surroundings In the analysis of perceived time (Table 5.38), design phase was a significant effect, F(2,42)=25.60, p<0.0001. Based on the multiple comparisons in Table 5.39, perceived time was significantly different in each design phase. 79 Table 5.38 Analysis of perceived time during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual G Residual I ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random 1 2 2 26 2 2 4 4 42 42 0.07 0.25 1.12 0.7236 25.60 0.88 1.73 0.66 0.6462 1.4664 0.7910 0.7824 0.3416 <0.0001*** 0.4225 0.1618 0.6240 Table 5.39 Comparisons of perceived time in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 5.07 0.935** Detailed Design 4.28 1.731** 0.796** 6.01 5.07 In the analysis of excessive work perceptions (Table 5.40), design phase was a significant effect, F(2,33)=9.73, p=0.0005 and the three way interaction was also significant, F(4,37)=2.96, p=0.0324. From the post hoc comparisons in Table 5.41, participants perceptions were significantly different in each phase. The multiple comparisons for the three-way interaction are provided in Appendix A.12. For groups with manual support, the perception of excessive work was lower in conceptual design than preliminary design. For individuals with manual support, the perception of excessive work was significantly different in each design phase. For individuals without support the perception was lower in detailed design compare to conceptual and preliminary design. Table 5.40 Analysis of excessive work during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual CD Residual PD Residual DD *p<0.05 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 1 2 2 29 2 2 4 4 17 1 28 0.72 0.60 0.33 0.6733 9.73 1.01 1.19 2.96 0.2350 0.0261 0.9214 0.4042 0.5533 0.7250 0.0005*** 0.3763 0.3298 0.0324* 80 Table 5.41 Multiple comparisons of excessive work in each design phase mean Conceptual Design Preliminary Design *p<0.05 **p<0.01 Preliminary Design 5.6 0.2222* Detailed Design 5.1 0.7872** 0.5650* 5.9 5.6 Challenge As reported in Appendix A.10, there were no significant differences in the variance analysis conducted on challenge. The questions associated with challenge were developing ability, level of interest in the problem, freedom in approaching the job, level of problem difficulty, and ability to see the results of work. In the analysis of ability (Table 5.42), the interaction between team design and project support was significant, F(2,29)=5.28, p=0.0111. Based on the multiple comparisons in Table 5.43, for groups, the mean ability was significantly higher in treatments without project support compared to treatments with automated support. For treatments without support, the mean ability was significantly higher for groups than individuals. These differences are shown in Figure 5.4. Table 5.42 Variance analysis of ability during the design process Source Effect DF Variance Component F value Probability Between TD PS TD*PS s/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual G Residual I *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random 1 2 2 29 2 2 4 4 39 39 3.67 0.36 5.28 0.3647 0.42 0.11 1.47 0.29 0.0343 0.1594 0.0655 0.6989 0.0111* 0.6606 0.8951 0.2296 0.8822 Table 5.43 Multiple comparisons of ability for the interaction between team design and project support mean Group automated Group manual Group none Individual automated Individual manual *p<0.05 **p<0.01 Group manual 5.13 0.3333 Group none 5.69 0.8889* 0.5556 Individual automated 5.33 0.5370 0.2037 0.3519 Individual manual 4.56 0.2407 0.5741 1.1296* 0.7778 Individual none 4.39 0.4074 0.7407 1.2963** 0.9444 0.1667 4.80 5.13 5.69 5.33 4.56 81 6 5 Mean Score 4 3 2 1 A AB C AB C AB C A Group Individual Automated Manual Project Support None Figure 5.4 Comparisons of developed ability perceptions for team design and project support In the analysis on the level of interest in the task (Table 5.44), design phase was significant, F(2,38)=6.78, p=0.0030. Based on the multiple comparisons in Table 5.45, the interest was rated significantly higher in conceptual and preliminary design compared to detailed design. Table 5.44 Variance analysis on task interest during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual G Residual I **p<0.01 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random 1 2 2 28 2 2 4 4 38 38 1.97 0.81 0.69 0.2187 6.78 0.18 0.38 0.77 0.2654 0.8278 0.1713 0.4546 0.5110 0.0030** 0.8401 0.8247 0.5546 Table 5.45 Multiple comparisons of the mean task interest in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 6.05 0.0185 Detailed Design 5.48 0.5463** 0.5648** 6.03 6.05 In the analysis on freedom in approaching the task (Table 5.46), design phase was significant, F(2,37)=7.79, p=0.0015. Based on the multiple comparisons in Table 5.47, there was significantly more freedom during conceptual design compared to preliminary and detailed design. 82 Table 5.46 Analysis of freedom during the design process Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual CD Residual PD Residual DD **p<0.01 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 1 2 2 45 2 2 4 4 3 18 26 1.52 0.97 0.09 0.3806 7.79 2.03 0.36 0.94 0.0937 0.3690 1.5985 0.2239 0.3872 0.9156 0.0015** 0.1454 0.8374 0.4518 Table 5.47 Multiple comparisons of freedom in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 5.72 0.3333** Detailed Design 5.41 0.6471** 0.3148 6.06 5.72 In the analysis on perceptions on the level of problem difficulty, shown in Table 5.48, design phase was significant, F(2,36)=3.48, p=0.0416. The problem was rated significantly more difficult during conceptual design compared to preliminary and detailed design as shown in Table 5.49. Table 5.48 Variance analysis on problem difficulty during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual CD Residual PD Residual DD *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 1 2 2 39 2 2 4 4 27 12 9 1.75 1.34 0.84 0.2534 3.48 2.95 0.78 1.00 1.0037 0.1559 0.1274 0.1933 0.2738 0.4390 0.0416* 0.0652 0.5468 0.4181 83 Table 5.49 Comparisons of problem difficulty between each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 5.84 0.4722 * Detailed Design 5.81 0.4444* 0.0887 5.37 5.84 From the analysis of the participants ability to see the results of their work (Table 5.50), design phase was significant, F(2,60)=8.26, p=0.0007. Based on the multiple comparisons in Table 5.51, the mean ratings were significantly different in each design phase. Table 5.50 Analysis of ability to see results during the design process Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS DP*S/TD*PS ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 1 2 2 30 2 2 4 4 60 0.04 0.09 2.35 0.4831 8.26 0.46 0.36 0.21 0.5130 0.8381 0.9145 0.1122 0.0007*** 0.6352 0.8375 0.9299 Table 5.51 Multiple comparisons of the mean ability to see results based on phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 5.07 0.935** Detailed Design 4.28 1.731** 0.796** 6.01 5.07 Resources As reported in Appendix A.10, there were no significant differences in the variance analysis conducted on resources. However, the responses to questions associated with resources were analyzed, including access to the appropriate equipment, access to information, and clearly defined responsibility. In the analysis of access to the right equipment (Table 5.52), the three-way interaction was significant, F(4,42)=3.30, p=0.0193. Refer to Appendix A.12 for the multiple comparisons table and figures. During conceptual design, in treatments without project support, individuals rated their access to the proper equipment higher than groups. Individuals without project support rated their access to the proper equipment higher during conceptual design than in detailed or preliminary design. 84 Table 5.52 Variance analysis on access to right equipment during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual CD Residual PD Residual DD *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 1 2 2 35 2 2 4 4 14 18 24 0.17 0.09 0.24 0.2201 0.79 1.59 0.28 3.30 0.2780 0.4251 0.9591 0.6861 0.9120 0.7881 0.4593 0.2170 0.8893 0.0193* There were no significant differences from the analysis conducted on access to information or clearly defined responsibility. Refer to Appendix A.10 for the variance summary tables. Reflective Participants were asked to reflect over the entire design process to evaluate the components of job satisfaction. A MANOVA was conducted on comfort, challenge, and resources to determine the affect on main effects and interactions, shown in Table 5.53 (the complete MANOVA table is in Appendix A.11). Table 5.53 MANOVA to test the affect of reflective job satisfaction components on main effects and interactions Source Team Design Project Support Team Design * Project Support F 0.565 0.123 0.520 P 0.642 0.993 0.791 A MANOVA was also conducted on the response to individual questions used to calculate comfort, challenge, and resources as shown in Table 5.53. The complete MANOVA table is provided in Appendix A.11. Table 5.54 MANOVA to test the affect of responses to all reflective questions on main effects and interactions Source Team Design Project Support Team Design * Project Support F 0.970 0.636 1.453 P 0.507 0.885 0.152 Comfort In the analysis on comfort, there were no significant effects (refer to Appendix A.10). The analysis on participants perceptions of excessive work, physical surroundings, personal problems and time were not significant. Refer to Appendix A.10 for the ANOVA summary tables. 85 Challenge In the analysis on challenge, there were no significant effects as shown in Appendix A.10. In the analysis on developing ability (Table 5.55), the interaction between team design and project support was significant, F(2,30)=4.76, p=0.016. From the comparisons in Table 5.56, for individuals, the mean ability was significantly higher in treatments with automated support compared to treatments without support. In treatments without support, developing ability was significantly higher for groups than individuals. Table 5.55 ANOVA for reflective developing ability Source TD PS TD*PS S/TD*PS Total *p<0.05 DF 1 2 2 30 35 SS 2.9660 0.1914 8.1914 25.7963 37.1451 MS 2.9660 0.0957 4.0957 0.8599 F 3.45 0.11 4.76 P 0.073 0.895 0.016* Table 5.56 Comparisons of reflective ability based on team design and project support mean Group automated Group manual Group none Individual automated Individual manual *p<0.05 **p<0.01 Group manual 5.33 0.3889 Group none 5.94 1.000 0.6111 Individual automated 5.50 0.5556 0.1667 0.4444 Individual manual 4.83 0.1111 0.5000 1.1111* 0.6667 Individual none 4.17 0.7778 1.1667* 1.7778** 1.3333* 0.6667 4.94 5.33 5.94 5.50 4.83 7 6 Mean Score 5 4 3 2 1 Automated Manual Project Support None AB AC A AC BC B Group Individual Figure 5.5 Comparing reflective developing ability for team design and project support In the analysis conducted on the perceptions on if the problem was interesting or difficult, if the participants had freedom in designing their work process, and the ability to see the results, there were no significant effects. The variance analysis tables are provided Appendix A.10. 86 Resources No significant effects were found from the analysis on resources. Similarly there were no significant effects in the analysis conducted on access to the appropriate equipment and information, and clearly defined responsibility. The variance analysis tables are provided Appendix A.10. 5.6 Supplemental Questionnaire Responses Participants responded to questions regarding aspects of design (doubt about work, ability to meet objectives, how well they liked the overall design, if they did the best that they could) and planning (satisfaction, ease of use, efficiency, and effectiveness of the tools). The complete list of questions is provided in Appendix A.2. 5.6.1 Design Related Questions Design Process In the variance analysis conducted on doubt in ability, shown in Table 5.57, design phase was significant, F(2,43)=3.93, p=0.0272. Doubt was significantly higher during conceptual design compared to detailed design as reported in Table 5.58. Table 5.57 Variance analysis on doubt during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS Residual G Residual I *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random 1 2 2 28 2 2 4 4 43 43 1.75 0.29 2.11 0.9855 3.91 2.51 1.65 0.20 0.3178 0.9379 0.1972 0.7542 0.1406 0.0276* 0.0931 0.1802 0.9361 Table 5.58 Multiple comparisons of doubt in each design phase mean Conceptual Design Preliminary Design *p<0.01 Preliminary Design 5.06 0.3148 Detailed Design 4.85 0.5185* 0.2037 5.37 5.06 Reflective The reflective questions included perceptions of how well participants liked the system they built, if they designed the best system possible, and how well they met their objectives. A MANOVA was conducted on supplemental design questions (Table 5.59). There were no significant differences due to effects in the analyses conducted on the supplemental design questions. Refer to Appendix A.10 for the ANOVA tables. 87 Table 5.59 MANOVA to test the affect of supplemental design questions on main effects and interactions Source F P Team Design Project Support Team Design * Project Support 0.700 2.232 0.702 0.560 0.054 0.649 5.6.2 Planning Planning and Tracking Related Questions A MANOVA was conducted on supplemental planning questions (Table 5.60). Questions included efficiency, effectiveness, productivity, ease, and satisfaction of using the support tools, doubt in ability to conduct the plan and created the best plan. The complete MANOVA is in Appendix A.11. Table 5.60 MANOVA to test the affect of supplemental planning questions Source Team Design Project Support Team Design * Project Support *p<0.05 F 1.288 1.264 3.200 P 0.324 0.335 0.030* There was a significant interaction between team design and project support in the analysis on doubt in ability to carry out the plan, F(1,20)=7.70, p=0.012, as reported in Table 5.61. Based on the post hoc analysis in Table 5.62, groups using manual support had more doubt than individuals using manual support or groups using automated support. Table 5.61 ANOVA for doubt in ability to satisfy the plan Source TD PS TD*PS S/TD*PS Total *p<0.05 DF 1 1 1 20 23 SS 3.375 0.227 6.3386 16.4633 26.4039 MS 3.375 0.227 6.339 0.823 F 4.10 0.28 7.70 P 0.056 0.605 0.012* Table 5.62 Comparisons of doubt for the interaction between team design and project support mean Individual Automated Individual Manual Group Automated *p<0.05 **p<0.01 Individual Manual 5.00 0.8333 Group Automated 4.44 0.2778 0.5555 Group Manual 3.22 0.9445 1.7778** 1.2233* 4.17 5.00 4.44 No differences were found for creating the best plan possible, ease of use, efficiency, effectiveness, productivity, and satisfaction during planning (shown in Appendix A.10). Design Process A MANOVA was conducted on the responses supplemental questions on the tracking tools asked after each design phase. The questions included ease of use, efficiency, effectiveness, productivity, and 88 satisfaction with the planning and tracking tools; refer to Table 5.63 for the MANOVA summary table. The complete MANOVA table is reported in Appendix A.11. Table 5.63 MANOVA to test the affect of responses to supplemental questions on tracking tools Source Team Design Project Support Design Phase Team Design * Project Support Team Design * Design Phase Project Support * Design Phase Team Design *Project Support * Design Phase F 2.18 0.89 1.49 2.47 1.03 1.15 0.65 P 0.1076 0.5107 0.1706 0.0768 0.4313 0.3442 0.7605 Design phase was significant in the analysis of ease of use (Table 5.64), F(2,40)=5.403, p=0.008. From the post hoc comparisons (Table 5.65), the tracking tools were rated easier to use during conceptual design than during preliminary and detailed design. Table 5.64 ANOVA for the project support tools ease of use of during design Source Effect DF Variance Component F value Probability Between TD PS TD*PS S/TD*PS Within DP DP*TD DP*PS DP*TD*PS DP*S/TD*PS **p<0.01 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 1 2 2 20 2 2 4 4 40 0.91 1.17 0.79 0.3417 5.40 1.43 2.66 0.64 0.5040 0.3520 0.2929 0.3843 0.0084** 0.2507 0.0825 0.5344 Table 5.65 Multiple comparisons for ease of use in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 4.97 0.5834** Detailed Design 4.97 0.5834** 0.0000 5.56 4.97 There were no significant effects in the analysis of the efficiency, effectiveness, productivity, or satisfaction with the project support tools, as reported in Appendix A.10. Reflective A MANOVA was conducted on responses to supplemental planning and tracking questions asked after the design process was complete. Questions included ability to stay on budget and schedule, productivity, and satisfaction. Refer to Table 5.66 for the MANOVA summary table. The complete MANOVA table is reported in Appendix A.11. In the analysis on productivity, satisfaction, ability to stay on schedule and ability to stay on budget, there were no significant differences (Appendix A.10). 89 Table 5.66 MANOVA to test the affect of reflective supplemental planning and tracking questions Source Team Design Project Support Team Design * Project Support F 0.095 0.603 0.452 P 0.983 0.666 0.769 5.7 Correlations Relationship between Job Satisfaction and Mental Workload The relationship between job satisfaction and mental workload were explored as shown in Table 5.67. In conceptual design, there was a significant positive correlation between mental demand and job satisfaction (r=0.376) and a negative correlation between job satisfaction and frustration (r=-0.358). In preliminary and detailed design there was a negative correlation with frustration (preliminary: r=-0.424; detailed: r=-0.513). Reflectively there was a negative relationship between job satisfaction and frustration (r=-0.433). Table 5.67 Correlation between job satisfaction/challenge and mental workload/frustration/TLX Factor Planning Job satisfaction Challenge Conceptual Design Job satisfaction Challenge Preliminary Design Job satisfaction Challenge Detailed Design Job satisfaction Challenge Reflective Job satisfaction Challenge *p<0.05 **p<0.01 Mental Demand 0.302 0.373 0.376* 0.381* 0.272 0.264 0.062 0.091 0.169 0.165 Frustration -0.320 -0.175 -0.358* -0.249 -0.424** -0.436** -0.513** -0.465** -0.433** -0.370* NASA TLX 0.017 0.202 0.055 0.048 0.225 0.194 -0.065 -0.071 0.026 0.064 Relationship between Job Satisfaction and Performance The relationship between job satisfaction and performance were explored as shown in Table 5.68. There was a significant positive correlation between job satisfaction and cost effectiveness, system effectiveness and reliability. Comfort and resources were the facets of job satisfaction which were significantly correlated with these performance measures. Materials cost and errors were significant negative correlations with the comfort and resource facets of job satisfaction. 90 Table 5.68 Correlation between job satisfaction and performance Factor Cost effectiveness System effectiveness Reliability Life-cycle cost Design cost Material cost Errors Scoping document Gantt chart *p<0.05 **p<0.01 Job Satisfaction 0.361* 0.386* 0.378* -0.201 0.194 -0.355* -0.419 -0.003 0.383 Comfort 0.387* 0.437** 0.378* -0.262 0.195 -0.427** -0.389* -0.117 0.423* Challenge 0.162 0.177 0.262 -0.072 0.151 -0.181 -0.349* 0.057 0.385 Resource 0.487** 0.483** 0.428** -0.227 0.197 -0.384* -0.437** 0.042 0.119 91 Chapter 6 Results for Role Data Previous analyses averaged data from groups for comparison to individual data. In this section the data for groups was analyzed with design role (designer, manufacturer, and purchaser) as a factor. Throughout this chapter the following abbreviations were used: role (R), designer (Des), manufacturer (Manf), purchaser (Pur), design phase (DP), conceptual design (CD), preliminary design (PD), detailed design (DD), project support (PS), automated (A or auto), manual (M), none (N), and ungrouped (U). Table 6.1 provides a summary of the variance analysis method and significant effects. Refer to Appendix A.9 for the goodness of fit data when the variable was with variance grouping. Table 6.1 Summary of data analysis amongst group members for variables during planning Dependent Variable Planning Analysis Method Significant Effects Significant Comparisons NASA TLX Mental demand Physical demand Temporal demand Performance Effort Frustration Job Satisfaction Comfort Excessive work Physical surroundings Perceived time Personal problems Challenge Develop ability Interesting problem Freedom Problem difficulty See results Resources Equipment Information Responsibility Competent members Helpful members Supplemental Questions Doubt Best Ease of use Efficiency Effectiveness Productivity Satisfaction *p<0.05 **p<0.01 ANOVA ANOVA Transform: Log10(x+1) ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA Transform: Log10(reflected x+1) ANOVA Var grouping: R Var grouping: PS ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA Var grouping: PS ANOVA ANOVA ANOVA ANOVA: Transform: Log10(reflected x+1) ANOVA ANOVA ANOVA PS: F(1,30)=6.96* PS: F(1,30)=4.26* A<M A<M PS: F(1,30)=6.32* PS*R: F(2,18)=3.76* A>M Manf: A>M PS: F(1,30)=7.38* PS: F(1,30)=10.54** PS: F(1,30)=8.90** PS: F(1,30)=5.69* PS: F(1,30)=6.15* R: F(2,30)=4.28* PS: F(1,30)=8.65** A>M A>M A>M A>M A>M Des<Manf A>M 92 Table 6.1 Summary of data analysis amongst group members for variables during planning (continued) Dependent Variable Planning Analysis Method Significant Effects Significant Comparisons Group workload Value group interaction ANOVA Difficulty of group ANOVA interaction Degree cooperation ANOVA Overall group workload ANOVA Group workload evaluated by outside observers Value of group interaction ANOVA Difficulty of group ANOVA interaction Degree of cooperation ANOVA Overall group workload ANOVA Critical Team Behaviors Total ineffective Transform: ( x + 1) + x Total effective Ineffective communication Effective communication Ineffective cooperation Effective cooperation Ineffective coordination Effective coordination Ineffective accept feedback Effective accept feedback Ineffective give feedback Effective give feedback Ineffective adaptability Effective adaptability Ineffective team spirit & morale Effective team spirit & morale Supplemental Observations Time Money Non-task *p<0.05 **p<0.01 PS: F(1,30)=6.00* PS: F(1,30)=10.81** A<M A<M ANOVA Logistic Transform: ( x + 1) + x ANOVA Transform: ( x + 1) + x Logistic ANOVA Transform: ( x + 1) + x PS: F(1,30)=5.03* A<M PS: F(1,30)=6.94* A<M ANOVA ANOVA ANOVA ANOVA PS: F(1,30)=6.27* R: F(2,30)=6.58** PS: F(1,30)=9.45** A<M Des=Manf<Pur A<M 93 Table 6.2 Summary of data analysis amongst group members for variables during design Dependent Variable Design Process Analysis Method Significant Effects Significant Comparisons NASA TLX Var grouping: U DP: F(2,90)=22.20*** DP*PS: F(4.90)=3.59** DP: F(2,54)=5.46** DP: F(2,57)=60.84*** DP: F(2,77)=23.72*** DP*PS: F(4,64)=3.05* DP: F(2,90)=7.55** DP: F(2,90)=14.11*** PS: F(2,45)=4.02* DP: F(2,90)=9.76* PS: F(2,45)=4.21* DP: F(2,90)=9.76*** DP*PS*R: F(8,90)=2.20* Mental Demand Physical Demand Temporal Demand Performance Effort Frustration Job Satisfaction Comfort Var grouping: DP Var grouping: PS Var grouping: PS Var grouping: PS Var grouping: U Var grouping: U Var grouping: U Var grouping: U Excessive work Var grouping: DP PS: F(2,53)=3.92* DP: F(2,68)=5.16** DP*PS*R: F(8,68)=3.22** Physical surroundings Perceived time Personal problems Challenge Develop ability Interesting Problem Var grouping: R Var grouping: DP Var grouping: R Var grouping: DP Var grouping: R Var grouping: DP PS: F(2,46)=6.07** DP: F(2,61)=21.28*** PS: F(2,48)=3.22* DP: F(2,55)=6.64** PS*R: F(4,58)=2.54* CD<PD=DD CD: A=M>N A: CD<PD=DD N: CD<PD=DD CD=PD>DD CD<PD=DD CD<PD=DD PD: M<N CD<PD=DD CD<PD=DD A=M<N CD>DD A=M<N CD>DD CD,A: D>P Des,A: CD>DD Des,N: CD>DD Manf,M: CD=PD>DD Pur,A: CD<PD Pur,M: CD<PD CD,Pur: A<N PD,Pur: M<N DD,Manf: M<N M<N CD>DD CD,A: Des=Manf>Pur PD,M: Des=Manf>Pur Des,N: CD>DD Manf,A: CD>DD Manf,M: CD=PD>DD Pur,A: CD>PD Pur,M: CD>PD CD,Pur: A<M=N PD,Pur: A=N>M A=M<N CD>PD=DD A<N CD=PD>DD A: Des>Pur M: Des<Manf=Pur Pur: A<M=N CD<PD=DD Freedom Problem difficulty See results *p<0.05 **p<0.01 ***p<0.001 Var grouping: DP Var grouping: PS Var grouping: U DP: F(2,90)=10.01*** 94 Table 6.2 Summary of data analysis amongst group members for variables during design (continued) Dependent Variable Design Process Analysis Method Significant Effects Significant Comparisons Resources Var grouping: U PS*R: F(4,45)=2.71* DP*PS*R: F(8,90)=2.15* A: Des>Pur Pur: A<M=N CD,A: Manf>Pur DD,A: Des>Pur DD,M: Des=Manf<Pur Manf,A: CD>DD Manf,M: CD=PD>DD CD,Pur: A<N PD,Pur: A<N DD,Manf: M<N DD,Pur: A<M=N Equipment Information Responsibility Competent members Var grouping: U Var grouping: R Var grouping: U Var grouping: DP PS: F(2,74)=3.69* PS*R: F(4,74)=3.19* DP: F(2,90)=6.681** PS*R: F(4,45)=3.326* Helpful members Transform: Log10 (reflected x+1) A=M<N A: Des>Pur M: Des<Pur Pur: A<M=N CD>DD A: Des>Pur M: Des<Manf=Pur Des: A=N>M Pur: A<M=N N>M=A CD>PD=DD Des=Pur<Manf CD>PD CD>PD A>M CD=PD>DD CD<PD=DD CD>PD=DD CD<PD=DD Supplemental Design Questions Doubt Var grouping: U Supplemental Planning Questions Ease of use Var grouping: U Efficiency Var grouping: U Effectiveness Var grouping: U Productive Var grouping: U Satisfaction Var grouping: U Group Workload Value of group interaction Var grouping: DP Difficulty of group Var grouping: U interaction Degree of cooperation Var grouping: DP Overall group workload Var grouping: U Group workload evaluated by outside observers Value of group interaction Var grouping: U Difficulty of group Var grouping: U interaction Degree of cooperation Var grouping: U Overall group workload Var grouping: U *p<0.05 **p<0.01 ***p<0.001 PS: F(2,45)=4.93* DP: F(2,90)=11.58*** R: F(2,30)=5.05* DP: F(2,60)=3.84* DP: F(2,60)=3.80* PS: F(1,30)=6.78* DP: F(2,57)=12.88*** DP: F(2,90)=4.73* DP: F(2,61)=11.39*** DP: F(2,90)=17.92*** DP: F(2,30)=10.13*** DP: F(2,30)=75.86*** CD<PD=DD CD<PD=DD 95 Table 6.2 Summary of data analysis amongst group members for variables during design (continued) Dependent Variable Design Process Analysis Method Significant Effects Significant Comparisons Critical Team Behaviors Total ineffective Total effective Ineffective communication Var grouping: DP Var grouping: DP Var grouping: DP R: F(2,58)=6.52** DP: F(2,62)=14.07*** R: F(2,59)=4.43* DP: F(2,58)=28.28*** DP*PS: F(4,69)=3.95** Des=Manf<Pur CD<PD=DD Des=Manf<Pur CD<PD=DD CD: A=M<N PD: A>N A: CD=DD<PD N: CD>PD=DD Des=Manf<Pur CD=PD>DD CD<PD=DD Effective communication Ineffective cooperation Effective cooperation Ineffective coordination Effective coordination Ineffective accept feedback Effective accept feedback Ineffective give feedback Effective give feedback Ineffective adaptability Effective adaptability Ineffective team spirit & morale Effective team spirit & morale Supplemental Observations Time Logistic Logistic Var grouping: DP Logistic Var grouping: DP Logistic Var grouping: DP Var grouping: R R: F(2,50)=3.58* DP: F(2,60)=10.61*** DP: F(2,58)=33.25*** DP: F(2,60)=24.02*** PS: F(2,57)=3.91* R: F(2,45)=6.92** DP: F(2,94)=15.57*** CD<PD=DD A<M=N Des=Manf<Pur CD<PD=DD Var grouping: DP Var grouping: DP R: F(2,54)=4.34* DP: F(2,57)=9.71*** PS*R: F(2,54)=3.14* R: F(2,54)=7.19** DP: F(2,58)=17.33*** PS*DP: F(4,64)=4.00** Des=Manf<Pur CD=DD<PD M: Des=Manf<Pur Pur: A=N<M Des=Manf<Pur CD=DD<PD CD: A<M=N PD: A<N DD: A=M>N A: CD<PD=DD M: CD<PD=DD N: CD=DD<PD CD=PD<DD A: CD=PD<DD Money Var grouping: DP Non-task *p<0.05 **p<0.01 ***p<0.001 Var grouping: DP DP: F(2,56)=8.44*** DP*PS: F(4,63)=3.72** 96 Table 6.3 Summary of the data analysis amongst group members for reflective variables Dependent Variable Reflective Analysis Method Significant Effects Significant Comparisons NASA TLX Mental demand Physical demand Temporal demand Performance Effort Frustration Job Satisfaction Comfort Excessive work Physical surroundings Perceived time Personal problems Challenge Develop ability Interesting problem Freedom Problem difficulty ANOVA Var grouping: R ANOVA Transform: Log10(reflected x) ANOVA ANOVA ANOVA ANOVA ANOVA Transform: Log10(reflected x+1) ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA ANOVA PS: F(2,45)=3.27* A<N PS: F(2,45)=4.259* PS*R: F(4,45)=2.930* A<N A: Des=Manf>Pur Pur: A<M=N A: Manf>Pur Pur: A<M=N PS*R: F(4,45)=2.634* See results Transform: Log10(reflected x+1) Resources ANOVA Equipment ANOVA Information Var grouping: R Responsibility ANOVA Competent members Transform: Log10(reflected x+1) Helpful members Var grouping: R Supplemental Design Questions Best ANOVA Liked Transform: Log10(reflected x+1) Met objectives ANOVA Supplemental Planning Questions On schedule ANOVA On budget ANOVA Ease of use ANOVA Productivity ANOVA Satisfaction ANOVA Group workload Value of group interaction Transform: Log10(reflected x+1) Difficulty of group interaction ANOVA Degree of cooperation Transform: Log10(reflected x+1) Overall group workload ANOVA Team workload evaluated by outside observers Value of group interaction ANOVA Difficulty of group interaction ANOVA Degree of cooperation ANOVA Overall group workload ANOVA *p<0.05 PS: F(1,30)=4.39* PS: F(1,30)=5.00* PS: F(1,30)=4.30* A>M A>M A>M PS: F(2,45)=3.30* A<N 97 6.1 6.1.1 NASA TLX Analysis of the NASA TLX for Groups by Role Inter-rater correlation coefficients were determined for the NASA TLX ratings by participants in groups and the averaged score. The coefficients ranged from a low of 0.3672 from the reflective analysis to a high of 0.6150 from the design process, as shown in Table 6.4. These results suggested that while an average might be sufficient to compare between the groups and individuals something more may be occurring indicating a need to conduct an analysis based on the roles that were played within the groups. Table 6.4 Reliability between the TLX for each group member and the mean TLX Variable Planning Design Reflective Correlation 0.5259 0.6150 0.3672 6.1.2 NASA TLX in Planning There were no significant differences in the analysis of the NASA TLX (as reported in Appendix A.13). Recall the NASA TLX was calculated by a weighted average of mental demand, physical demand, temporal demand, performance, effort, and frustration. A MANOVA was conducted on the components of the NASA TXL, as shown in Table 6.5. The complete MANOVA table is reported in Appendix A.14. Table 6.5 MANOVA to test the affect of NASA TLX components during planning Source Project Support Role Project Support * Role *p<0.05 F 3.129 0.681 0.436 P 0.020* 0.761 0.941 There were no significant differences in the analysis of mental demand, physical demand, performance, and frustration due to treatments. In the analysis of temporal demand (Table 6.6), project support was significant, F(1,30)=6.96, p=0.013. Participants with automated support (mean=10.2, sd=5.01) rated temporal demand significantly lower than those with manual support (mean=14.2, sd=3.895). Table 6.6 ANOVA for temporal demand during planning Source PS R PS*R S/PS*R Total *p<0.05 DF 1 2 2 30 35 SS 149.38 33.89 6.87 644.08 834.23 MS 149.38 16.94 3.44 21.47 F 6.96 0.79 0.16 P 0.013* 0.463 0.853 Project support was significant in the analysis of effort, F(1,30)=4.26, p=0.048 (Table 6.7). Participants with manual support (mean=12.1, sd=3.706) rated effort significantly higher than participants with automated support (mean=9.1, sd=4.75). 98 Table 6.7 ANOVA for effort during planning Source PS R PS*R S/PS*R Total *p<0.05 DF 1 2 2 30 35 SS 79.27 49.53 9.85 558.49 697.14 MS 79.27 24.77 4.93 18.62 F 4.26 1.33 0.26 P 0.048* 0.280 0.769 6.1.3 NASA TLX in the Design Process In the analysis of the NASA TLX (Table 6.8), design phase, F(2,90)= 22.20, p<0.0001, and the interaction between design phase and project support, F(4,90)=3.59, p=0.009, were significant. From the comparisons between each design phase (Table 6.9), the NASA TLX was significantly lower in conceptual design than the other phases. From the analysis on the interaction (Table 6.10 and Figure 6.1), in conceptual design, treatments without support were significantly lower than treatments with project support. For treatments with automated support, the NASA TLX was significantly higher in preliminary and detailed design compared to conceptual design. For treatments without project support, the mean NASA TLX score was lower in conceptual design than the other phases. Table 6.8 Variance analysis for the NASA TLX Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/PS*R **p<0.01 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 45 2 4 4 8 90 0.15 0.53 1.16 1.8656 22.20 3.59 0.48 1.50 2.6335 0.8605 0.5897 0.3428 <0.0001*** 0.0091** 0.7519 0.1677 Table 6.9 Multiple comparisons of the NASA TLX in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 14.62 1.6217** Detailed Design 14.94 1.9406** 0.3189 13.00 14.62 99 Table 6.10 Comparisons of the NASA TLX based on design phase and project support mean CD auto CD manual CD none PD auto PD manual PD none DD auto DD manual *p<0.05 13.46 13.64 11.90 14.71 14.45 14.72 14.76 14.64 CD manual 13.64 0.1867 CD none 11.90 1.5533* 1.7400* PD auto 14.71 1.2489* 1.0622 2.8022* PD manual 14.45 0.9906 0.8039 2.5439* 0.2583 PD none 14.72 1.2589 1.0722 2.8122* 0.0100 0.2683 DD auto 14.76 1.2989* 1.1122 2.8522* 0.0500 0.3083 0.0400 DD manual 14.64 1.1794 0.9928 2.7328* 0.0694 0.1889 0.0794 0.1194 DD none 15.43 1.9767* 1.7900* 3.5300* 0.7278 0.9861 0.7178 0.6778 0.7972 20 16 A Mean Rating 12 8 4 0 Automated Manual Project Support None CD CD AC AC AC DD B AC D D CD PD DD Figure 6.1 Comparing mean TLX ratings for the interaction between design phase and project support A MANOVA was conducted on the rating scales used to determine the NASA TLX and is reported in Table 6.11. The complete MANOVA table is reported in Appendix A.14. Table 6.11 MANOVA to test the affect of NASA TLX components on main effects and interactions Source Project Support Role Design Phase Project Support * Role Project Support * Design Phase Role * Design Phase Project Support * Role * Design Phase ***p<0.001 F 0.43 0.44 15.61 0.54 1.14 0.70 0.92 P 0.9442 0.9393 <0.0001*** 0.9558 0.2994 0.8499 0.6179 In the analysis of mental demand, design phase was significant, F(2,54)=5.46, p=0.0069, as reported in Table 6.12. From the post hoc comparisons in Table 6.13, mental demand was higher in conceptual and preliminary design than in detailed design. 100 Table 6.12 Variance analysis for mental demand Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R Residual CD Residual PD Residual DD **p<0.01 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 48 2 4 4 8 24 17 35 0.06 0.53 0.78 6.3256 5.46 0.67 0.55 1.36 7.1970 5.4806 16.5753 0.9454 0.5908 0.5417 0.0069** 0.6154 0.6966 0.2319 Table 6.13 Multiple comparisons of mental demand in each design phase mean Conceptual Design Preliminary Design *p<0.05 **p<0.01 Preliminary Design 14.7 0.4326 Detailed Design 12.9 2.1819** 1.7493* 15.1 14.7 Design phase was significant, F(2,57)=60.84, p<0.0001) in the analysis on physical demand (Table 6.14). From the multiple comparisons given in Table 6.15, the perception of physical demand was significantly lower during conceptual design compared to preliminary and detailed design. Table 6.14 Variance analysis for physical demand Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R Residual CD Residual PD Residual DD ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 78 2 4 4 8 3 39 40 0.60 0.06 0.72 5.5892 60.84 0.87 1.35 1.34 1.5589 26.5816 30.9766 0.5539 0.9398 0.5841 <0.0001*** 0.4895 0.2629 0.2374 101 Table 6.15 Multiple comparisons of physical demand in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 7.46 5.6054** Detailed Design 8.30 6.4530** 0.8476 1.85 7.46 In the analysis of temporal demand, design phase was significant, F(2,90)= 25.030, p<0.0001 (Table 6.16). Temporal demand was significantly lower during conceptual design compared to preliminary or detailed design as shown in Table 6.17. Table 6.16 Variance analysis for temporal demand Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R Residual A Residual M Residual N ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 3 2 4 4 8 31 36 35 0.13 0.79 1.19 2.4277 23.72 1.73 0.20 0.66 22.2272 23.0623 8.6755 0.8807 0.4605 0.3308 <0.0001*** 0.1549 0.9369 0.7270 Table 6.17 Multiple comparisons of temporal demand in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 14.45 3.9476** Detailed Design 15.95 5.4407** 1.4931 10.50 14.45 Shown in Table 6.18, the interaction between design phase and project support was significant in the analysis of performance ratings, F(4,64)=3.05, p=0.0232. From the comparisons (Table 6.19), in treatments with manual support, performance ratings were significantly higher during conceptual design than in preliminary design. In preliminary design, the performance rating was higher in treatments with manual support compared to treatments without project support as shown in Figure 6.2. 102 Table 6.18 Variance analysis for performance ratings Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R Residual A Residual M Residual N *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 5 2 4 4 8 32 37 28 0.19 0.31 0.69 1.7109 0.17 3.05 0.53 0.74 15.8005 6.0081 6.3914 0.8252 0.7382 0.6011 0.8402 0.0232* 0.7162 0.6520 Table 6.19 Comparison of performance ratings for design phase and project support mean 13.32 14.75 13.09 14.86 12.71 14.62 13.37 13.73 CD manual 14.75 1.4339 CD none 13.09 0.2289 1.6628 PD auto 14.86 1.5456 0.1117 1.7744 PD manual 12.71 0.6050 2.0389* 0.3761 2.1506 PD none 14.62 1.3022 0.1317 1.5311 0.2433 1.9072* DD auto 13.37 0.0550 1.3789 0.2839 1.4906 0.6600 1.2472 DD manual 13.73 1.1839 1.0178 0.6450 1.1294 1.0211 0.8861 0.3611 DD none 14.68 1.3644 0.0694 1.5933 0.1811 1.9694* 0.0622 1.3094 0.9483 CD auto CD manual CD none PD auto PD manual PD none DD auto DD manual *p<0.05 16 AB AB AB A B AB A AB A 12 Mean 8 Rating 4 0 Automated Manual Project Support Figure 6.2 Comparing performance ratings for the interaction between design phase and project support CD PD DD None 103 In the analysis of effort, design phase was significant, F(2,90)=7.55, p=0.0009 (Table 6.20). From the post hoc analysis in Table 6.21, the perception was that less effort was required in conceptual design than in either preliminary design or detailed design. Table 6.20 Variance analysis for effort ratings Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/PS*R ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 45 2 4 4 8 90 0.49 1.36 1.09 6.6650 7.55 1.85 0.41 0.71 8.7112 0.6185 0.2662 0.3743 0.0009*** 0.1258 0.8043 0.6826 Table 6.21 Multiple comparisons of in effort each design phase mean Conceptual Design Preliminary Design *p<0.05 **p<0.01 Preliminary Design 13.51 1.2933* Detailed Design 14.41 2.1963** 0.9030 12.21 13.51 Design phase was significant, F(2,90)=14.11, p<0.0001 in the analysis on frustration as shown in Table 6.22. From the multiple comparisons, frustration was significantly lower in conceptual design compared to preliminary and detailed design (refer to Table 6.23). Table 6.22 Variance analysis for frustration Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/PS*R ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 45 2 4 4 8 90 1.21 0.22 0.14 9.5597 14.11 1.40 0.28 0.63 21.2226 0.3068 0.7996 0.9674 <0.0001*** 0.2402 0.8871 0.7482 104 Table 6.23 Multiple comparisons of frustration in each design phase mean Conceptual Design Preliminary Design **p<0.001 Preliminary Design 9.76 3.9291** Detailed Design 10.05 4.2135** 0.2844 5.83 9.76 6.1.4 Reflective NASA TLX No significant differences were found in the analysis for the reflective NASA TLX. Refer to Appendix A.13 for the variance table. A MANOVA was conducted on mental demand, physical demand, temporal demand, performance, effort, and frustration, which is reported in Table 6.24. The complete MANOVA table is reported in Appendix A.14. Table 6.24 MANOVA to test the affect of reflective TLX components on main effects and interactions Source Project Support Role Project Support * Role F 0.763 0.703 0.778 P 0.686 0.744 0.760 No significant differences were found in the analyses on mental demand, temporal demand, performance, effort, and frustration. Refer to Appendix A.13 for the summary ANOVA tables. In the analysis of physical demand (Table 6.25), project support was significant, F(2,45)=3.27, p=0.047. From the multiple comparisons in Table 6.26, physical demand was significantly lower in treatments with automated support than in treatments without support. Table 6.25 ANOVA for the reflective physical demand Source PS R PS*R s/PS*R Total *p<0.05 DF 2 2 4 45 53 SS 176.29 3.46 20.95 1214.76 1415.46 MS 88.15 1.73 5.24 26.99 F 3.27 0.06 0.19 P 0.047* 0.938 0.940 Table 6.26 Multiple comparisons between types of project support for reflective physical demand mean Automated Manual *p<0.05 4.03 7.45 Manual 7.45 3.4211 None 8.17 4.1423* 0.7212 6.2 Job Satisfaction Planning There were no significant effects in the analysis of job satisfaction during planning. The ANOVA summary table is located in Appendix A.13. A MANOVA was conducted on comfort, challenge and resources, the facets of job satisfaction, which is summarized in Table 6.27. The complete MANOVA table is reported in Appendix A.14. 105 Table 6.27 MANOVA to test the affect of job satisfaction components during planning Source Project Support Role Project Support * Role F 0.96 0.10 1.30 P 0.425 0.996 0.274 A MANOVA was conducted on questions used determine job satisfaction, which is summarized in Table 6.28. The complete MANOVA table is reported in Appendix A.14. Table 6.28 MANOVA to test the affect of responses to job satisfaction questions on main effects and interactions during planning Source Project Support Role Project Support * Role F 2.044 1.440 0.521 P 0.081 0.160 0.940 Comfort There were no significant effects in the analysis of comfort as reported in Appendix A.13. In the analysis on time perception (Table 6.29), project support was significant, F(1,30)=6.32, p=0.018. Participants with automated support (mean=5.889, sd=0.832) had significantly higher scores compared to those with manual support (mean=4.611, sd=1.883). Table 6.29 ANOVA for perceived time during planning (transformed: Log10(reflected x+1)) Source PS R PS*R S/PS*R Total *p<0.05 DF 1 2 2 30 35 SS 0.2545 0.0340 0.1088 1.2076 1.6049 MS 0.2545 0.0170 0.0544 0.0403 F 6.32 0.42 1.35 P 0.018* 0.659 0.274 The following did not reveal significant effects: excessive work, develop ability, physical surroundings, and personal problems. The ANOVA summary tables are located in Appendix A.13. Challenge In the variance analysis for challenge (shown in Table 6.30), the interaction between project support and role was significant, F(2,18)=3.76, p=0.043. From the multiple comparisons in Table 6.31, manufacturers with automated support had significantly higher challenge scores than manufacturers with manual support. These results are also shown graphically in Figure 6.3. Table 6.30 ANOVA for challenge during planning Source Effect DF Variance Component F value Probability PS R PS*R Residual D Residual M Residual P *p<0.05 Fixed Fixed Fixed Random Random Random 1 2 2 10 10 10 2.31 0.00 3.76 6.7500 7.0167 41.2167 0.147 1.000 0.043* 106 Table 6.31 Multiple comparisons of challenge between project support and role mean Automated designer Automated manufacturer Automated purchaser Manual designer Manual manufacturer *p<0.05 **p<0.01 Automated manufacturer 26.83 2.833 Automated purchaser 23.67 0.333 3.167 Manual designer 23.50 0.500 3.333* 0.167 Manual manufacturer 20.67 3.333* 6.167** 3.000 2.833 Manual purchaser 23.83 0.167 3.000 0.167 0.333 3.167 24.00 26.83 23.67 23.50 20.67 31 26 21 Mean Score 16 11 6 1 AB BC A C ABC ABC Automated Manual Designer Manufacturer Role Purchaser Figure 6.3 Comparisons of challenge for the interaction between project support and role From the analysis on developing ability, interest and difficulty of the problem, freedom in approaching the work, and ability to see the results, there were no significant effects. Refer to Appendix A.13 for the variance analysis tables. Resources There were no significant effects in the analysis of resources. Similarly, no significant effects were found from the analysis on access to the appropriate equipment, information, responsibility, and competent and helpful group members. The ANOVA summary tables are located in Appendix A.13. Design Process In the analysis of job satisfaction during design (refer to Table 6.32), design phase and project support were significant (design phase: F(2,90)=3.25, p=0.0433; project support: F(2,45)=4.02, p=0.0248). As reported in Table 6.33, job satisfaction was significantly higher in conceptual design than in detailed design. From the post hoc comparisons on project support (Table 6.34), job satisfaction was higher in treatments without project support than in treatments with project support. 107 Table 6.32 Variance analysis for job satisfaction Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/(PS*R) *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 45 2 4 4 8 90 4.02 0.20 1.71 49.5185 3.25 0.32 1.33 1.07 29.8469 0.0248* 0.8175 0.1633 0.0433* 0.8616 0.2659 0.3900 Table 6.33 Multiple comparisons of job satisfaction in each design phase mean Conceptual Design Preliminary Design *p<0.05 Preliminary Design 79.56 0.7037 Detailed Design 77.67 2.5925* 1.8889 80.26 79.56 Table 6.34 Multiple comparisons of job satisfaction for each project support level mean Automated Manual *p<0.05 Manual 76.74 0.6481 None 83.35 5.9630* 6.6111* 77.39 76.74 A MANOVA was conducted on the facets of job satisfaction, which is summarized in Table 6.35. The complete MANOVA table is reported in Appendix A.14. Table 6.35 MANOVA to test the affect of job satisfaction facets on main effects and interactions Source Project Support Role Design Phase Project Support * Role Project Support * Design Phase Role * Design Phase Project Support * Role * Design Phase *p<0.05 ***p<0.001 F 1.47 0.33 4.44 1.54 0.63 1.16 1.66 P 0.2052 0.9192 0.0004*** 0.1302 0.8114 0.3183 0.0286* A MANOVA was conducted on the responses to job satisfaction questions, which is summarized in Table 6.36. The complete MANOVA table is reported in Appendix A.14. 108 Table 6.36 MANOVA to test the affect of job satisfaction responses on main effects and interactions Source Project Support Role Design Phase Project Support * Role Project Support * Design Phase Role * Design Phase Project Support * Role * Design Phase *p<0.05 ***p<0.001 F 1.71 1.27 4.75 1.13 0.87 0.92 1.22 P 0.0488* 0.2249 <0.0001*** 0.3051 0.7276 0.6351 0.0971 Comfort In the analysis of comfort, summarized in Table 6.37, design phase, F(2,90)=9.76, p=0.0001, project support, F(2,45)=4.21, p=0.021, and the three-way interaction, F(8,90)=2.20, p=0.0344, were significant. From the post hoc analysis on the effect of design phase (Table 6.38), comfort was higher in conceptual design than in detailed design. Comfort was higher in treatments without project support compared to treatments with project support as shown in Table 6.39. The multiple comparisons for the three-way interaction are summarized in Appendix A.15. During conceptual design in treatments with automated support, designers rated comfort significantly higher than purchasers. Designers using automated support and designers without support had significantly higher comfort scores in conceptual design compared to detailed design. For manufacturers with manual support, comfort was significantly higher in conceptual design and preliminary design compared to detailed design. For purchasers using automated or manual support, comfort was significantly higher in conceptual design than in preliminary design. During conceptual design, purchasers without project support rated comfort significantly higher than those with automated support. While during preliminary design, purchasers without project support rated comfort higher than those with manual support. Table 6.37 Variance analysis for comfort Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/(PS*R) *p<0.05 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 45 2 4 4 8 90 4.21 0.14 0.30 6.4815 9.76 0.62 1.89 2.20 6.0693 0.0210* 0.8690 0.8777 0.0001*** 0.6521 0.1184 0.0344* 109 Table 6.38 Multiple comparisons of comfort in each design phase mean Conceptual Design Preliminary Design *p<0.05 Preliminary Design 22.24 1.0741 Detailed Design 21.41 1.9074* 0.8333 23.31 22.24 Table 6.39 Multiple comparisons of comfort for project support levels mean Automated Manual *p<0.05 **p<0.01 Manual 21.37 0.3148 None 23.91 2.2222** 2.5370* 21.69 21.37 The variance analysis conducted on excessive work (Table 6.40), indicated design phase, project support and the three way interaction were significant (design phase: F(2,68)=5.16, p=0.0087; project support: F(2,53)=3.92, p=0.0258; and design phase * project support * role: F(8,68)=3.22, p=0.0037). From the comparisons (Table 6.41), conceptual design was rated significantly higher than detailed design. Treatments with manual support had a significantly lower rating than those without support (Table 6.42). Appendix A.15 contains a summary of multiple comparisons for the three-way interaction. In conceptual design, purchasers with automated support had significantly lower ratings than the others with automated support. In preliminary design, purchasers with manual support had significantly lower ratings than the others with manual support. In detailed design treatments with manual support, manufacturers had significantly lower ratings than designers. In treatments without project support, designers had lower ratings in detailed design than conceptual design; a similar result occurred for manufacturers with automated support. Manufacturers with manual support had significantly lower ratings in detailed design than the other design phases. Purchasers with automated project support had significantly higher ratings in preliminary design than conceptual design. Purchasers with manual project support had significantly lower ratings in preliminary design than conceptual design. In conceptual design, purchasers with automated support had significantly lower ratings than the other purchasers. In preliminary design, purchasers with manual support had significantly lower ratings than the other purchasers. Table 6.40 Variance analysis for excessive work in groups Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R Residual CD Residual PD Residual DD *p<0.05 **p<0.01 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 53 2 4 4 8 27 21 35 3.92 0.93 1.24 0.2351 5.16 2.37 1.48 3.22 0.5611 0.5147 1.6627 0.0258* 0.4005 0.3052 0.0087** 0.0619 0.2194 0.0037** 110 Table 6.41 Multiple comparisons of excessive work in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 5.69 0.2778 Detailed Design 5.33 0.6296** 0.3519 5.96 5.69 Table 6.42 Multiple comparisons of excessive work in each project support level mean Automated Manual *p<0.05 Manual 5.31 0.3519 None 6.00 0.3333 0.6852** 5.67 5.31 In the analysis of physical surroundings (Table 6.43), project support was significant, F(2,46)=6.07, p=0.0045. From the comparisons in Table 6.44, physical surroundings were rated significantly lower in treatments with project support compared to treatments without support. Table 6.43 Variance analysis for physical surroundings Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R Residual D Residual M Residual P **p<0.01 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 46 2 4 4 8 30 32 30 6.07 0.11 0.35 0.8352 0.02 1.21 0.82 1.68 0.0931 0.3427 0.4739 0.0045** 0.8946 0.8424 0.9799 0.3136 0.5184 0.1217 Table 6.44 Comparisons of physical surroundings based on project support mean Automated Manual *p<0.05 **p<0.01 Manual 5.1 0.4074 None 6.2 0.7037* 1.1111** 5.5 5.1 Design phase was significant in the analysis on perceived time (Table 6.45), F(2,90)=21.28, p<0.0001. From the post hoc analysis (Table 6.46), the mean rating was significantly higher in conceptual design compared to preliminary and detailed design. 111 Table 6.45 Variance analysis for perceived time Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R Residual CD Residual PD Residual DD ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 63 2 4 4 8 24 33 39 1.29 2.61 0.86 0.2863 21.28 0.40 1.49 1.45 0.7395 1.7407 3.5203 0.2827 0.0814 0.4940 <0.0001*** 0.8056 0.2142 0.1909 Table 6.46 Multiple comparisons of perceived time in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 4.98 1.0370** Detailed Design 4.5 1.5185** 0.4815 6.02 4.98 There were no significant effects in the analysis of personal problems. The variance analysis summary table is located in Appendix A.13. Challenge There were no significant effects in the analysis of challenge. Refer to Appendix A.13 for the variance analysis summary table. In the analysis of ability, Table 6.45, project support was significant, F(2,48)=3.22, p=0.0490. The mean rating was significantly higher in treatments without support compared to treatments with automated support (Table 6.46). Table 6.47 Variance analysis for developing ability Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R Residual D Residual M Residual P *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 48 2 4 4 8 28 33 32 3.22 1.28 1.00 0.9585 1.01 1.50 0.53 1.40 0.2647 0.7533 0.5152 0.0490* 0.2884 0.4173 0.3670 0.2085 0.7138 0.2122 112 Table 6.48 Multiple comparisons of developing ability based on project support mean Automated Manual *p<0.05 Manual 5.1 0.3333 None 5.7 0.8889* 0.5556 4.8 5.1 Design phase and the interaction between project support and role were significant in the analysis on the level of interest in the task as shown in Table 6.49 (design phase: F(2,55)=6.64, p=0.0026; project support*role: F(4,58)=2.54, p=0.0496). From the post hoc analysis (Table 6.50), the rating was significantly higher in conceptual and preliminary design than in detailed design. From the multiple comparisons (summarized in Table 6.51 and Figure 6.4), in treatments with automated support, the mean rating was significantly higher for designers than purchasers. In treatments with manual support, the mean rating was significantly higher for purchasers than designers and manufacturers. For purchasers, the mean rating was significantly higher in treatments with manual support and without support compared to treatments with automated support. Table 6.49 Variance analysis for interest perceptions Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R Residual CD Residual PD Residual DD *p<0.05 **p<0.01 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 58 2 4 4 8 12 39 26 2.28 0.54 2.54 0.3238 6.64 0.38 0.28 0.72 0.2033 1.5204 0.4307 0.1114 0.5873 0.0496* 0.0026** 0.8246 0.8882 0.6725 Table 6.50 Multiple comparisons of perceived interest in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 6.2 0.0185 Detailed Design 5.6 0.6481** 0.6296** 6.2 6.2 113 Table 6.51 Multiple comparisons of interest for the interaction between project support and role mean Auto Design Auto Manf Auto Purch Manual Design Manual Manf Manual Purch None Design None Manf *p<0.05 **p<0.01 Auto Manf 5.8 0.444 Auto Purch 5.3 0.889* 0.444 Manual Design 5.6 0.611 0.167 0.278 Manual Manf 5.6 0.611 0.167 0.278 0.000 Manual Purch 6.6 0.333 0.778 1.222** 0.944* 0.944* None Design 6.2 0.056 0.389 0.833 0.556 0.556 0.389 None Manf 6.2 0.000 0.444 0.889* 0.611 0.611 0.333 0.056 None Purch 6.5 0.278 0.722 1.167** 0.889* 0.889* 0.056 0.333 0.278 6.2 5.8 5.3 5.6 5.6 6.6 6.2 6.2 7 6 5 Mean 4 Score 3 2 1 AC AC ACD BD ABCD ABCD BD ACD B Automated Manual None Designer Manufacturer Role Purchaser Figure 6.4 Comparison of level of interest in the problem based on project support and role From Table 6.52, in the analysis of ability to see results, design phase was significant, F(2,90)=10.01, p=0.0001. The scores were significantly higher during preliminary and detailed design compared to conceptual design as shown in Table 6.55. Table 6.52 Variance analysis for ability to see results Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/PS*R ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 45 2 4 4 8 90 1.71 1.00 1.01 0.5889 10.01 0.33 0.78 0.58 0.9679 0.1930 0.3751 0.4111 0.0001*** 0.8539 0.5382 0.7890 114 Table 6.53 Multiple comparisons of ability to see results in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 5.63 0.6481** Detailed Design 5.78 0.7963** 0.1481 4.98 5.63 In the analysis on freedom and problem, there were no significant effects or interactions. The variance summary table is provided in Appendix A.13. Resources In the analysis of resources (Table 6.54), the interaction between project support and role and the three-way interaction were significant (project support * role: F(4,45)= 2.71, p=0.0416; design phase * project support * role: F(8,90)=2.15, p=0.0397). The post hoc analysis of the two way interaction is shown in Table 6.55 and Figure 6.5. In treatments with automated support, designers had significantly higher resource ratings than purchasers. Purchasers rated their resources significantly lower in treatments with automated support compared to treatments with manual support and without support. The multiple comparisons for the three-way interaction are reported in Appendix A.15. During conceptual design with automated support, manufacturers rated their resources significantly higher than purchasers. During detailed design with automated support, designers had significantly higher ratings than purchasers. In treatments with manual support, purchasers rated their resources significantly higher than designers and manufacturers. Designers with manual support had significantly higher ratings during conceptual design than in preliminary design. Manufacturers with automated support had significantly lower ratings in detailed design than conceptual design. Manufacturers with manual support rated resources significantly lower in detailed design than conceptual and preliminary design. During each design phase, purchasers with automated support rated resources significantly lower than purchasers without project support. Purchasers during detailed design had significantly higher ratings in treatments with manual support than in treatments with automated support. Manufacturers during detailed design had significantly lower ratings in treatments with manual support than treatments without project support. Table 6.54 Variance analysis for resources Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/(PS*R) *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 45 2 4 4 8 90 2.93 0.22 2.71 6.7741 2.75 0.96 1.95 2.15 4.4444 0.0634 0.8015 0.0416* 0.0692 0.4337 0.1094 0.0387* 115 Table 6.55 Multiple comparisons of resources based on project support and role Auto manf 29.22 0.6111 Auto purch 26.11 3.7222* 3.1111 Manual design 27.61 2.2222 1.6111 1.5000 Manual manf 27.67 2.1667 1.5556 1.5556 0.0556 Manual purch 29.94 0.1111 0.7222 3.8333* 2.3333 2.2778 None design 29.94 0.1111 0.7222 3.8333* 2.3333 2.2778 0.0000 None manf 29.28 0.5556 0.0556 3.1667 1.6667 1.6111 0.6667 0.6667 None purch 32.00 2.1667 2.7778 5.8889** 4.3889* 4.3333* 2.0556 2.0556 0.5458 mean Auto design Auto manf Auto purch Manual design Manual manf Manual purch None design None manf *p<0.05 **p<0.01 29.83 29.22 26.11 27.61 27.67 29.94 29.94 29.28 36 31 26 Mean 21 Score 16 11 6 1 AC D AB CD C AB CD AB CD AB B CD D Automated Manual None Designer Manufacturer Role Purchaser Figure 6.5 Comparisons of resources for the interaction between project support and role In the analysis of equipment, information and responsibility, there were no significant effects or interactions. Refer to Appendix A.13 for the variance analysis summary table. As shown in Table 6.56, project support, F(2,74)=3.69, p=0.0026, and the interaction between project support and role, F(4,74)=3.19, p=0.0496, were significant in the analysis on perceptions of group member competency. From the comparisons (Table 6.57), ratings were significantly higher in treatments without project support than in treatments with project support. In treatments with automated support, the rating was significantly higher for designers than purchasers, as shown in Table 6.58 and Figure 6.6. In treatments with manual support, the rating was significantly higher for purchasers than designers and manufacturers. For purchasers, the rating was significantly higher in treatments with manual support and without supported compared to treatments with automated support. 116 Table 6.56 Variance analysis for group member competence Source Effect DF Variance Component F value Probability Between PS R PS*R s/PS*R Within DP DP*PS DP*R DP*PS*R Residual CD Residual PD Residual DD *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 74 2 4 4 8 7 37 34 3.69 0.49 3.19 0.2423 3.07 0.40 0.86 1.63 0.1132 0.7904 1.1467 0.0296* 0.6145 0.0179* 0.0541 0.8071 0.4930 0.1339 Table 6.57 Multiple comparisons of group member competence based on project support mean Automated Manual *p<0.05 Manual 5.8 0.0370 None 6.4 0.5185* 0.5556* 5.9 5.8 Table 6.58 Multiple comparisons of group member competence based on project support and role Auto Manf 6.1 0.056 Auto Purch 5.3 0.833* 0.778 Manual Design 5.4 0.722 0.667 0.111 Manual Manf 5.7 0.500 0.444 0.333 0.222 Manual Purch 6.4 0.222 0.278 1.056** 0.944* 0.722 None Design 6.1 0.056 0.000 0.778 0.667 0.444 0.278 None Manf 6.4 0.222 0.278 1.056** 0.944* 0.722 0.000 0.278 None Purch 6.7 0500 0.556 1.333** 1.222** 1.000* 0.278 0.556 0.278 mean Auto Design Auto Manf Auto Purch Manual Design Manual Manf Manual Purch None Design None Manf *p<0.05 **p<0.01 6.2 6.1 5.3 5.4 5.7 6.4 6.1 6.4 117 7 AC 6 5 Mean 4 Score 3 2 1 Designer BC ABC ABC AB AC B CD AD Automated Manual None Manufacturer Role Purchaser Figure 6.6 Comparison of group member competence based on project support and role Project support and the interaction between project support and role were significant in the analysis on perceptions of group member helpfulness shown in Table 6.59 (project support * role: F(4,45)=3.326, p=0.018; design phase: F(2,90)=6.681, p=0.002). From the post hoc comparisons (Table 6.60), group members were rated significantly more helpful during conceptual design compared to detailed design. From the multiple comparisons summarized in Table 6.61 and Figure 6.7, in treatments with automated support, designers rated members significantly more helpful than purchasers. Purchasers rated members significantly more helpful compared to designers and manufacturers in treatments with manual support. Designers rated members significantly more helpful in treatments with automated support and without support compared to treatments with manual support. Purchasers rated members significantly more helpful in treatments without support and with manual support compared to treatments with automated support. Table 6.59 Variance analysis for group member helpfulness (transformed: Log10 (reflected x+1)) Source Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/PS*R Total *p<0.05 **p<0.01 DF SS MS F P 2 2 4 45 2 4 4 8 90 0.1500 0.0643 0.4370 1.4790 0.1300 0.0048 0.0112 0.0513 0.8740 0.07491 0.03214 0.10900 0.03288 0.06491 0.00120 0.00279 0.00641 0.00972 2.279 0.978 3.326 6.681 0.124 0.288 0.660 0.114 0.384 0.018* 0.002** 0.974 0.885 0.726 118 Table 6.60 Multiple comparisons of group member helpfulness in each design phase mean tmean1 Conceptual Design Preliminary Design 6.2 0.43 5.9 0.47 Preliminary Design 5.9 0.47 0.0364 Detailed Design 5.6 0.50 0.0693** 0.0330 **p<0.01 1 tmean refers to the mean of the transformed data Table 6.61 Multiple comparisons of helpful group members based on project support and role mean tmean Auto Design Auto Manf Auto Purch Manual Design Manual Manf Manual Purch None Design None Manf *p<0.05 **p<0.01 6.2 0.44 5.7 0.49 5.1 0.57 5.3 0.54 5.5 0.53 6.5 0.39 6.1 0.45 5.9 0.46 Auto Manf 5.7 0.49 0.0512 Auto Purch 5.1 0.57 0.1337** 0.0826 Manual Design 5.3 0.54 0.1029* 0.0517 0.0308 Manual Manf 5.5 0.53 0.0934* 0.0422 0.0403 0.0095 Manual Purch 6.5 0.39 0.0489 0.1001* 0.1826** 0.1518** 0.1423** None Design 6.1 0.45 0.0139 0.0373 0.1198** 0.0890* 0.0795 0.0628 None Manf 5.9 0.46 0.0230 0.0282 0.1107* 0.0799 0.0704 0.0719 0.0091 None Purch 6.6 0.37 0.0615 0.1127* 0.1953** 0.1644** 0.1549** 0.0126 0.0754 0.0845 7 A 6 BD 5 Mean 4 Score 3 2 1 ACE AB BD E ACD B C C Automated Manual None Designer Manufacturer Role Purchaser Figure 6.7 Comparisons of helpfulness for the interaction between project support and role 119 Reflective In the analysis conducted on reflective job satisfaction, there were no significant effects (Appendix A.13). A MANOVA was conducted on comfort, challenge, and resources, which is summarized in Table 6.62. The complete MANOVA table is reported in Appendix A.14. There were no significant effects in the analyses conducted on comfort, challenge, and resources. Refer to Appendix A.13 for the ANOVA summary tables. Table 6.62 MANOVA to test affect of reflective job satisfaction facets on main effects and interactions Source Project Support Role Project Support * Role F 0.717 0.871 1.319 P 0.637 0.520 0.216 A MANOVA was conducted on the questions used to calculate comfort, challenge, and resources, which is summarized in Table 6.63. The complete MANOVA table is reported in Appendix A.14. Table 6.63 MANOVA to test the affect of reflective job satisfaction and supplemental design questions Source Project Support Role Project Support * Role F 1.070 0.677 0.904 P 0.403 0.887 0.672 Comfort In the analysis of the responses to questions used to determine comfort there were no significant differences. The ANOVA summary tables are located in Appendix A.13. Challenge Project support was significant in the analysis conducted on developing ability, F(2,45)=4.259, p=0.020, as shown in Table 6.64. From the multiple comparisons (Table 6.65), there were significantly lower ratings in treatments with automated supported compared to treatments without support. Table 6.64 Variance analysis for reflective developing ability Source Between PS R PS*R S/PS*R Total *p<0.05 DF SS MS F P 2 2 4 45 54 9.148 2.815 4.741 48.333 65.037 4.574 1.407 1.185 1.074 4.259 1.310 1.103 0.020* 0.280 0.367 Table 6.65 Multiple comparisons of reflective developing ability based on project support mean Automated Manual **p<0.01 Manual 5.3 0.3889 None 5.9 1.0000** 0.6111 4.9 5.3 120 The interaction between project support and role was significant, F(4,45)=2.930, p=0.031 (Table 6.66) in the analysis on reflective interest. From the multiple comparisons (summarized in Table 6.67 and Figure 6.8), treatments with automated support had significantly higher mean ratings for designers than purchasers. Purchasers had significantly higher mean ratings in treatments with manual support and without support compared to treatments with automated support. Table 6.66 Variance analysis for reflective interest in the problem Source PS R PS*R S/PS*R Total *p<0.05 DF 2 2 4 45 53 SS 2.926 0.148 8.074 31.000 42.148 MS 1.463 0.074 2.019 0.689 F 2.124 0.108 2.930 P 0.131 0.898 0.031* Table 6.67 Comparisons of reflective interest in the problem based on project support and role Mean 6.5 6.2 5.2 5.7 6.2 6.5 6.5 6.3 Auto Manf 6.2 0.3333 Auto Purch 5.2 1.3333** 1.0000* Manual Design 5.7 0.8333 0.5000 0.5000 Manual Manf 6.2 0.3333 0.0000 1.0000* 0.5000 Manual Purch 6.5 0.0000 0.3333 1.3333** 0.8333 0.3333 None Design 6.5 0.0000 0.3333 1.3333** 0.8333 0.3333 0.0000 None Manf 6.3 0.1667 0.1667 1.1667* 0.6667 0.1667 0.1667 0.1667 None Purch 6.7 0.1667 0.5000 1.5000** 1.0000* 0.5000 0.1667 0.1667 0.3333 Auto Design Auto Manf Auto Purch Manual Design Manual Manf Manual Purch None Design None Manf *p<0.05 **p<0.01 7 6 5 Mean Score 4 3 2 1 C AC AC AC AC AC AC B AC Automated Manual None Designer Manual Role Purchaser Figure 6.8 Comparison of reflective interest for project support and role 121 In the analysis on reflective problem difficulty, Table 6.68, the interaction between project support and role was significant, F(4,45)=2.634, p=0.046. Manufacturers had significantly higher ratings compared to purchasers in treatments with automated support, (Table 6.69 and Figure 6.9). For purchasers, the ratings were significantly higher for treatments with manual support and without support compared to treatments with automated support. Table 6.68 Variance analysis for reflective problem difficulty Source PS R PS*R S/PS*R Total *p<0.05 DF 2 2 4 45 53 SS 1.444 1.333 6.556 28.000 37.333 MS 0.722 0.667 1.639 0.622 F 1.161 1.071 2.634 P 0.322 0.351 0.046* Table 6.69 Multiple comparisons of reflective problem difficulty based on project support and role mean 5.7 6.3 4.8 5.5 5.7 6.0 5.8 6.0 Auto Manf 6.3 0.6667 Auto Purch 4.8 0.8333 1.500** Manual Design 5.5 0.1667 0.8333 0.6667 Manual Manf 5.7 0.0000 0.6667 0.8333 0.1667 Manual Purch 6.0 0.3333 0.3333 1.1667* 0.5000 0.3333 None Design 5.8 0.1667 0.5000 1.0000* 0.3333 0.1667 0.1667 None Manf 6.0 0.3333 0.3333 1.1667* 0.5000 0.3333 0.0000 0.1667 None Purch 6.2 0.5000 0.1667 1.3333** 0.6667 0.5000 0.1667 0.3333 0.1667 Auto Design Auto Manf Auto Purch Manual Design Manual Manf Manual Purch None Design None Manf *p<0.05 **p<0.01 7 6 5 Mean 4 Score 3 2 1 Designer Manual Role Purchaser AC AC C A A AC A A A Automated Manual None Figure 6.9 Comparison of reflective problem difficulty for project support and role 122 In the analysis on freedom and ability to see results there were no significant effects as reported in Appendix A.13. Resources There were no significant effects in the analysis on access to the right equipment, information, and responsibility, and group member competence and helpfulness as shown in Appendix A.13. 6.3 Supplemental Design Questions Design Process Perception of doubt was the supplemental design question. Because the rating scale was similar to that for job satisfaction questions, the variable was included in the MANOVA presented in Table 6.36. Project support, F(2,45)=4.93, p=0.0116, and design phase, F(2,90)=11.58, p<0.0001, were significant in the analysis on doubt (Table 6.70). Participants without support had significantly lower ratings compared to participants with project support as shown in Table 6.72. Doubt was significantly lower in conceptual design than other design phases (Table 6.71). Table 6.70 Variance analysis for doubt Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/PS*R *p<0.05 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 45 2 4 4 8 90 4.93 0.16 0.82 1.0716 11.58 1.03 0.72 0.88 0.9802 0.0116* 0.8498 0.5173 <0.0001*** 0.3964 0.5819 0.5378 Table 6.71 Multiple comparisons of doubt in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 5.17 0.6852** Detailed Design 4.98 0.8704** 0.1852 5.85 5.17 Table 6.72 Multiple comparisons of mean doubt based on project support mean Automated Manual *p<0.05 **p<0.01 Manual 4.81 0.3159 None 6.02 0.8519* 1.2037** 5.17 4.81 123 Reflective The MANOVA reported in Table 6.63 included both the supplemental reflective design questions and the job satisfaction questions since the measuring scales were similar. There were no significant differences in the analyses on did the best to their ability, liked the system and met objectives as shown in Appendix A.13. 6.4 Supplemental Planning Questions Planning A MANOVA was conducted on supplemental planning questions as shown in Table 6.28. Project support was significant, F(1,30)=7.38, p=0.011, in the analysis on doubt in ability to complete the project as planned (Table 6.73). For project support, participants with automated support (mean=4.444, sd=1.199) had significantly higher ratings than those with manual support (mean=3.222, sd=1.478). Table 6.73 ANOVA for doubt in ability during planning Source PS R PS*R S/PS*R Total *p<0.05 DF 1 2 2 30 35 SS 13.444 0.667 6.222 54.667 75.000 MS 13.444 0.333 3.111 1.822 F 7.38 0.18 1.71 P 0.011* 0.834 0.199 Project support was significant, F(1,30)=10.54, p=0.003 in the analysis of creating the best plan possible (Table 6.74). Participants with automated support (mean=4.833, sd=1.098) had significantly higher ratings during planning than participants with manual support (mean=3.556, sd=1.199). Table 6.74 ANOVA for the question, creating the best plan within their ability Source PS R PS*R S/PS*R Total **p<0.01 DF 1 2 2 30 35 SS 14.694 2.722 0.389 41.833 59.639 MS 14.694 1.361 0.194 1.394 F 10.54 0.98 0.14 P 0.003** 0.388 0.870 As shown in Table 6.75, project support was significant in the analysis of planning tools ease of use, F(1,30)=11.02, p=0.002. The participants using automated support (mean=5.778, sd=0.943) had a significantly higher mean ratings than those with manual support (mean=4.556, sd=1.542). Table 6.75 ANOVA for the ease of use during planning Source PS R PS*R S/PS*R Total **p<0.01 DF 1 2 2 30 35 SS 13.444 8.667 1.556 45.333 69.000 MS 13.444 4.333 0.778 1.511 F 8.90 2.87 0.51 P 0.006** 0.073 0.603 124 In the analysis of efficiency during planning, project support was significant, F(1,30)=5.69, p=0.024, as reported in Table 6.76. The participants with automated support rated efficiency higher (mean=5.8, sd=1.200) than those with manual support (mean=4.8, sd=1.654). Table 6.76 ANOVA for efficiency during planning (transformed: Log10(reflected x+1)) Source PS R PS*R S/PS*R Total *p<0.05 DF 1 2 2 30 35 SS 0.2562 0.1953 0.0576 1.3511 7.4246 MS 0.2562 0.09763 0.02879 0.04504 F 5.69 2.17 0.64 P 0.024* 0.132 0.535 There were no significant effects or interactions in the analysis of effectiveness during planning, as reported in Appendix A.13. In the analysis of productivity, project support, F(1,30)=6.15, p=0.019, and role, F(2,30)=4.28, p=0.023, were significant, as shown in Table 6.77. Those with automated support (mean=5.556, sd=0.922) had significantly higher mean ratings than those with manual support (mean=4.611, sd=1.1461). Manufacturers rated productivity significantly higher than designers (Table 6.78). Table 6.77 ANOVA for productivity during planning Source PS R PS*R s/PS*R Total *p<0.05 DF 1 2 2 30 35 SS 8.028 11.167 0.389 39.167 58.750 MS 8.028 5.583 0.194 1.306 F 6.15 4.28 0.15 P 0.019* 0.023* 0.862 Table 6.78 Multiple comparisons between roles for productivity during planning mean 4.5 5.8 Manufacturer 5.8 1.3333** Purchaser 4.9 0.4167 0.9167 Designer Manufacturer *p<0.05 Project support was the significant effect in the analysis on satisfaction, F(1,30)=8.65, p=0.006, as reported in Table 6.79. Participants with automated support (mean=5.389, sd=1.092) rated satisfaction significantly higher than those with manual support (mean=4.222, sd=1.309). Table 6.79 ANOVA for satisfaction during planning in groups Source PS R PS*R S/PS*R Total **p<0.01 DF 1 2 2 30 35 SS 12.250 6.722 0.167 42.500 61.639 MS 12.250 3.361 0.083 1.417 F 8.65 2.37 0.06 P 0.006** 0.111 0.943 125 Design Process A MANOVA was conducted on the supplemental planning questions, including ease of use, efficiency, effectiveness, productivity, and satisfaction. A summary of the MANOVA is provided in Table 6.80 and the complete MANOVA is reported in Appendix A.14. Table 6.80 MANOVA to test the affect of supplemental planning questions on main effects and interactions Source Project Support Role Design Phase Project Support * Role Project Support * Design Phase Role * Design Phase Project Support * Role * Design Phase F 1.39 1.25 1.73 0.64 1.45 1.23 0.67 P 0.2593 0.2956 0.0879 0.7711 0.1751 0.2467 0.8510 In the analysis of ease of use (Table 6.81), design phase and role were significant (phase: F(2,60)=3.84, p=0.0268; role: F(2,30)=5.05, p=0.0129). The comparisons between design phases (Table 6.82) indicated the mean ease of use was significantly higher in conceptual design than preliminary design. Manufacturers rated ease of use significantly higher than purchasers or designers (Table 6.83). Table 6.81 Variance analysis for ease of use Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/PS*R *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 30 2 2 4 4 60 2.75 5.05 0.35 0.8833 3.84 2.39 0.28 0.40 0.5852 0.1076 0.0129* 0.7042 0.0268* 0.1004 0.8867 0.8110 Table 6.82 Multiple comparisons of ease of use in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 4.78 0.500** Detailed Design 5.03 0.2500 0.2500 5.28 4.78 Table 6.83 Multiple comparisons of ease of use based on role mean Designer Manufacturer *p<0.05 **p<0.01 Manufacturer 5.81 1.1667** Purchaser 4.64 0.0000 1.1667* 4.64 5.81 126 There were no significant effects in the analysis of efficiency. Table 6.84 contains the summary of the variance analysis for the planning support effectiveness; design phase was significant, F(2,60)=3.80, p=0.0278. From the multiple comparisons in Table 6.85, effectiveness was significantly higher in conceptual design than preliminary design. Table 6.84 Variance analysis for effectiveness Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/PS*R *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 30 2 2 4 3 60 0.48 0.60 0.51 0.9833 3.80 0.08 1.41 0.89 0.8056 0.4923 0.5544 0.6040 0.0278* 0.9228 0.2422 0.4751 Table 6.85 Multiple comparisons of the mean effectiveness in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 4.17 0.5833** Detailed Design 4.47 0.2778 0.3056 4.75 4.17 There were no significant effects in the analysis of productivity (Appendix A.13). Project support was significant in the analysis of satisfaction, F(2,30)=6.78, p=0.0142, as shown in Table 6.86. The satisfaction with manual support (mean=4.11, sd=1.3270) was significantly lower than the satisfaction with automated support (mean=4.89, sd=1.0218). Table 6.86 Variance analysis for satisfaction Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/PS*R *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 30 2 2 4 4 60 6.78 2.26 0.74 0.4981 1.10 1.10 0.35 0.53 0.9130 0.0142* 0.1217 0.4863 0.3410 0.3410 0.8431 0.7123 Reflective A MANOVA was conducted on reflective supplemental planning questions as reported in Table 6.87. The complete MANOVA table is reported in Appendix A.14. 127 Table 6.87 MANOVA to test the affect of the reflective supplemental planning questions Source Project Support Role Project Support * Role F 1.889 1.171 0.909 P 0.141 0.334 0.489 There were no significant effects in the analysis on ability to remain on schedule and within budget as shown in Appendix A.13. Project support was significant in the analysis of the ease of use, F(1,30)=4.39, p=0.045 (Table 6.88). Participants using automated support (mean=5.667, sd=0.686) had higher mean ratings than those using manual support (mean=5.000, sd=1.188). Table 6.88 ANOVA for reflective ease of use Source PS R PS*R S/PS*R Total *p<0.05 DF 1 2 2 30 35 SS 4.0000 3.5000 1.1667 27.333 36.000 MS 4.0000 1.7500 0.5833 0.9111 F 4.39 1.92 0.64 P 0.045* 0.164 0.534 As shown in Table 6.89, project support was significant in the analysis on reflective productivity, F(1,30)=5.00, p=0.033. Participants with automated support (mean=5.278, sd=0.826) rated productivity significantly higher than those with manual support (mean=4.500, sd=1.339). Table 6.89 ANOVA on reflective productivity Source PS R PS*R S/PS*R Total *p<0.05 DF 1 2 2 30 35 SS 5.444 5.056 4.389 32.667 47.556 MS 5.444 2.528 2.194 1.089 F 5.00 2.32 2.02 P 0.033* 0.116 0.151 Project support was significant in the analysis of reflective satisfaction, F(1,30)=4.30, p=0.047 (Table 6.90). Participants with automated support (mean=5.556, sd=1.199) had significantly higher ratings than those with manual support (mean=4.667, sd=1.283). Table 6.90 ANOVA on reflective satisfaction Source PS R PS*R S/PS*R Total *p<0.05 DF 1 2 2 30 35 SS 7.111 1.056 1.722 49.667 59.556 MS 7.111 0.528 0.861 1.656 F 4.30 0.32 0.52 P 0.047* 0.729 0.600 128 6.5 Group Workload Planning A MANOVA was conducted on group workload scales from planning as shown in Table 6.91. The complete MANOVA table is reported in Appendix A.14. There were no significant effects or interactions found in analyses conducted on the value of group interaction, difficulty of group interaction, degree cooperation, and overall group workload. The summary tables are reported in Appendix A.13. Table 6.91 MANOVA to test the affect of group workload scales on main effects and interactions during planning Source Project Support Role Project Support * Role *p<0.05 F 0.485 1.358 0.642 P 0.747 0.237 0.729 Design Process A MANOVA was conducted on the group workload rating scales administered after each design phase as shown in Table 6.92. The complete MANOVA table is reported in Appendix A.14. Table 6.92 MANOVA to test the affect of group workload scales on main effects and interactions Source Project Support Role Design Phase Project Support * Role Project Support * Design Phase Role * Design Phase Project Support * Role * Design Phase *p<0.05 F 0.37 0.92 11.15 0.89 1.18 0.92 0.80 P 0.9334 0.5069 <0.0001*** 0.5776 0.2860 0.5457 0.7658 In the analysis on the value of group interaction, shown in Table 6.93, design phase was significant, F(2,57)=12.88, p<0.0001. From the multiple comparisons (Table 6.94), the value of the interaction was rated significantly lower during detailed design compared to the other design phases. 129 Table 6.93 Variance analysis for the value of group interaction Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R Residual CD Residual PD Residual DD ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 60 2 4 4 8 13 30 38 0.10 0.52 0.95 5.4888 12.88 1.38 2.25 1.48 4.5001 12.4503 31.5318 0.9064 0.5970 0.4441 <0.0001*** 0.2521 0.0738 0.1820 Table 6.94 Multiple comparisons of value of group interaction in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 15.59 0.8919 Detailed Design 12.35 4.1337** 3.2419** 16.48 15.59 Design phase was a significant effect in the analysis of the difficulty of group interaction, F(2,90)=4.73, p=0.0112, as reported in Table 6.95. From the multiple comparisons in Table 6.96, difficulty was significantly higher during preliminary and detailed design compared to conceptual design. Table 6.95 Variance analysis for the difficulty of group interaction Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R DP*S/PS*R *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 45 2 4 4 8 90 0.58 0.39 0.32 8.6419 4.73 1.87 0.13 0.86 21.4024 0.5663 0.6806 0.8616 0.0112* 0.1227 0.9691 0.5546 Table 6.96 Multiple comparisons of difficulty of group interaction in each design phase mean Conceptual Design Preliminary Design *p<0.05 **p<0.01 Preliminary Design 7.74 2.6033** Detailed Design 7.17 2.0341* 0.5693 5.14 7.74 130 As shown in Table 6.97, design phase was significant in the analysis on the degree of cooperation, F(2,61)=11.39, p<0.0001. Multiple comparisons are reported in Table 6.98. The degree of cooperation was rated significantly higher during conceptual design compared to the other design phases. Table 6.97 Variance analysis for degree of cooperation Source Effect DF Variance Component F value Probability Between PS R PS*R s/PS*R Within DP DP*PS DP*R DP*PS*R Residual CD Residual PD Residual DD ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 62 2 4 4 8 23 34 40 0.86 0.32 1.22 2.1259 11.39 1.83 1.57 1.44 5.2905 14.9848 25.9438 0.4273 0.7238 0.3130 <0.0001*** 0.1326 0.1917 0.1960 Table 6.98 Multiple comparisons of degree of cooperation in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 14.41 2.2328** Detailed Design 13.74 2.9013** 0.6685 16.64 14.41 Design phase was a significant effect in the analysis of the overall group workload, F(2,90)=17.92, p<0.0001 (Table 6.99). From the multiple comparisons in Table 6.100, the overall group workload was rated significantly higher during preliminary and detailed design compared to conceptual design. Table 6.99 Variance analysis for overall group workload Source Effect DF Variance Component F value Probability Between PS R PS*R s/PS*R Within DP DP*PS DP*R DP*PS*R DP*s/PS*R ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random 2 2 4 45 2 4 4 8 90 0.13 1.17 0.26 4.9208 17.92 0.86 0.75 0.73 12.5570 0.8766 0.3208 0.9016 <0.0001*** 0.4892 0.5576 0.6661 131 Table 6.100 Multiple comparisons of the mean overall group workload based on phase mean Conceptual Design Preliminary Design *p<0.05 Preliminary Design 13.31 2.9017** Detailed Design 14.35 3.9378** 1.0361 10.41 13.31 Reflective A MANOVA was conducted on the reflective group workload as shown in Table 6.101. Refer to Appendix A.14 for the complete MANOVA. There were no significant effects or interactions found in the ANOVA conducted on the reflective value of group interaction, difficulty of group interaction, and degree of cooperation. The variance summary tables are reported in Appendix A.13. Table 6.101 MANOVA to test the affect of reflective group workload scales Source Project Support Role Project Support * Role F 1.565 1.747 0.972 P 0.148 0.100 0.489 There was a significant main effect in the analysis conducted on the overall group workload shown in Table 6.102; project support was significant, F(2,45)=3.30, p=0.046. From post hoc comparisons (refer to Table 6.), the overall group workload was rated significantly lower in the treatments with automated support than in the treatments without support. Table 6.102 ANOVA for the reflective overall group workload Source PS R PS*R S/PS*R Total *p<0.05 DF 2 2 4 45 53 SS 85.55 38.28 73.37 582.54 779.74 MS 42.77 19.14 18.34 12.95 F 3.30 1.48 1.42 P 0.046* 0.239 0.244 Table 6.103 Multiple comparisons between project support types for overall group workload mean 11.8 13.9 Manual 13.9 2.1394 None 14.7 2.9922* 0.8528 Automated Manual *p<0.05 6.6 Group Workload Evaluated by Outsider Observers Planning A MANOVA was conducted on the group workload scales assess by outside observers during planning. The results are reported in Table 6.104. Refer to Appendix A.14 for the complete MANOVA table. In the analysis of the reflective value of group interaction, the difficulty of group interaction, the degree of cooperation, and overall team workload, there were no significant effects as reported in Appendix A.14. 132 Table 6.104 MANOVA to test the group workload scales assessed by outside observers during planning Source Project Support F 0.452 P 0.769 Design Process A MANOVA was conducted on the group workload assessed by outside observers during design. The results are reported in Table 6.105 (refer to Appendix A.14 for the complete MANOVA). Table 6.105 MANOVA to test the group workload assessed by outside observers on the main effects and interactions during design Source Project Support Design Phase Project Support * Design Phase ***p<0.001 F 0.79 21.42 1.46 P 0.6184 <0.0001*** 0.1541 There were no significant effects in the analysis of the value of group interaction or the degree of cooperation. Refer to Appendix A.4 for the variance summary table. Design phase was significant in the analysis on the difficulty of group interaction evaluated by outside observers, F(2,30)=10.13, p=0.0004 (Table 6.106). From the post hoc analysis in Table 6.107, the difficulty of group interaction was significantly higher during preliminary and detailed design compared to conceptual design. Table 6.106 ANOVA for difficulty of group interaction evaluated by outside observers Source Effect DF Variance Component F value Probability Between PS S/PS Within DP PS*DP S*DP/PS ***p<0.001 Fixed Random Fixed Fixed Random 2 15 2 4 30 0.34 1.5337 10.13 0.92 2.3430 0.7166 0.0004** 0.4662 Table 6.107 Comparisons of difficulty of group interaction evaluated by external observers mean Conceptual Design Preliminary Design *p<0.05 **p<0.01 8.03 10.19 Preliminary Design 10.19 2.1589** Detailed Design 9.79 1.7593* 0.3996 As reported in Appendix A.4, the degree of cooperation did not have significant effects. However, design phase was a significant effect in the analysis of the overall group workload evaluated by outside observers, F(2,30)=75.86, p<0.0001 (Table 6.108). From the comparisons in Table 6.109, overall group workload was significantly higher during preliminary and detailed design compared to conceptual design. 133 Table 6.108 ANOVA for overall group workload evaluated by outside observers Source Effect DF Variance Component F value Probability Between PS S/PS Within DP PS*DP S*DP/PS ***p<0.001 Fixed Random Fixed Fixed Random 2 15 2 4 30 1.13 1.5151 75.86 2.58 2.330 0.3484 <0.0001*** 0.0572 Table 6.109 Comparisons of overall group workload evaluated by external observers mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 13.86 4.6686** Detailed Design 15.15 5.9552** 1.2865 9.19 13.86 Reflective A MANOVA was conducted on reflective group workload assessed by outside observers. The results are reported in Table 6.110. Appendix A.14 contains the complete MANOVA table. In the analysis of the reflective value of group interaction, difficulty of group interaction, degree of cooperation, and overall group workload, there were no significant effects as reported in Appendix A.14. Table 6.110 MANOVA for reflective group workload scales assessed by outside observers Source Project Support F 0.401 P 0.908 6.7 6.7.1 Critical Team Behaviors Exploring Effective and Ineffective Behaviors One way to consider the critical team behavior data is to evaluate the percentage of effective and ineffective behaviors, which are summarized in Table 6.111 for each design phase and was the standard way to present the data historically. The percentage of behaviors across design phase showed that in most cases there were more effective behaviors observed than ineffective behaviors. Communication was an exception in all design phases and adaptability was an exception in conceptual design. However when the number of observations was considered for adaptability during conceptual design, the difference was inconclusive because the observers did not agree that an observation occurred. 134 Table 6.111 Percentage of effective and ineffective behaviors summarized by design phase Variable Total effective Total ineffective Accepting feedback Effective Ineffective Adaptability Effective Ineffective Communication Effective Ineffective Cooperation Effective Ineffective Coordination Effective Ineffective Giving feedback Effective Ineffective Team spirit & morale Effective Ineffective Total (number and percentage within each behavior) 1009 (88%) 138 (12.0%) 15 (85.7%) 2.5 (14.3%) 48.5 (93.3%) 3.5 (6.7%) 16 (27.4%) 42.5 (72.6%) 381.5 (93.8%) 25 (6.2%) 337 (89.0%) 41.5 (11.0%) 174 (96.1%) 7 (3.9%) 37 (69.8%) 16 (30.2%) Conceptual Design 232.5 (92.3%) 19.5 (7.7%) 0 (0%) 1 (100%) 0.0 (0%) 0.5 (100%) 4 (22.2%) 14 (77.8%) 136.5 (100%) 0 (0%) 55 (93.2%) 4 (6.8%) 27 (100%) 0 (0%) 10 (100%) 0 (0%) Preliminary Design 423.5 (85.9%) 69.5 (14.1%) 6.0 (80%) 1.5 (20%) 22 (88%) 3 (12%) 6 (25%) 18 (75%) 157.5 (91.8%) 14 (8.2%) 149 (87.9%) 20.5 (12.1%) 67 (97.1%) 2 (2.9%) 16 (62.7%) 9.5 (37.3%) Detailed Design 353 (87.8%) 49 (12.2%) 9 (100%) 0 (0%) 26.5 (100%) 0.0 (0%) 6 (36.4%) 10.5 (63.6%) 87.5 (88.8%) 11 (11.2%) 133 (88.4%) 17.5 (11.6%) 80 (94.1%) 5 (5.9%) 11 (62.9%) 6.5 (37.1%) The percentage of behaviors for each type of project support is shown in Table 6.112. During planning, the vast majority of behaviors were effective. Twice the amount of ineffective behaviors was observed in treatments with manual support than in treatments with automated support. In all cases except for one, there were more effective behaviors observed than ineffective behaviors. The exception was communication in treatments with manual support. The percentage of observations followed similar trends during the design process. During design, all levels of project support (automated, manual and unsupported) had more ineffective communication observations than effective observations. 135 Table 6.112 Percentage of effective and ineffective behaviors summarized by project support level Variable Planning Total effective Total ineffective Accepting feedback Effective Ineffective Adaptability Effective Ineffective Communication Effective Ineffective Cooperation Effective Ineffective Coordination Effective Ineffective Giving feedback Effective Ineffective Team spirit & morale Effective Ineffective Design Process Total effective Total ineffective Accepting feedback Effective Ineffective Adaptability Effective Ineffective Communication Effective Ineffective Cooperation Effective Ineffective Coordination Effective Ineffective Giving feedback Effective Ineffective Team spirit & morale Effective Ineffective Total (number and percentage within each behavior) 267 (89.4%) 31.5 (10.6%) 5.5 (84.6%) 1 (15.4%) 10 (100%) 0 (0%) 4.5 (31%) 10 (69%) 76 (86.9%) 11.5 (13.1%) 92 (96.3%) 3.5 (3.7%) 63 (94.7%) 3.5 (5.3%) 16 (88.9%) 2 (11.1%) 1009 (88%) 138 (12.0%) 15 (85.7%) 2.5 (14.3%) 48.5 (93.3%) 3.5 (6.7%) 16 (27.4%) 42.5 (72.6%) 381.5 (93.8%) 25 (6.2%) 337 (89.0%) 41.5 (11.0%) 174 (96.1%) 7 (3.9%) 37 (69.8%) 16 (30.2%) Automated 95.5 (94.1%) 6 (5.9%) 0.5 (100%) 0 (0%) 2.5 (100%) 0 (0%) 1 (66.7%) 0.5 (33.3%) 29.5 (88.1%) 4 (11.9%) 32.5 (95.6%) 1.5 (4.4%) 22.5 (100%) 0 (0%) 7 (100%) 0 (0%) 331 (87.2%) 48.5 (12.8%) 8.5 (94.4%) 0.5 ( 5.6%) 7.5 (93.75%) 0.5 (6.25%) 5 (21.7%) 18 (78.3%) 106.5 (95.9%) 4.5 (4.1%) 133.5 (88.4%) 17.5 (11.59%) 58 (95.9%) 2.5 (4.1%) 12 (70.6%) 5 (29.4%) Manual 171.5 (87.1%) 25.5 (12.9%) 5 (83.3%) 1 (16.7%) 7.5 (100%) 0 (0%) 3.5 (26.9%) 9.5 (73.1%) 46.5 (86.1%) 7.5 (13.9%) 59.5 (96.7%) 2 (3.3%) 40.5 (92%) 3.5 (8%) 9 (81.8%) 2 (18.2%) 345 (88.8%) 43.5 (11.2%) 3.5 (77.8%) 1.0 (22.2%) 19.5 (100%) 0.0 (0%) 3.5 (24.1%) 11 (75.9%) 142.5 (94.1%) 9 (5.9%) 105 (88.2%) 14 (11.8%) 58.5 (95.9%) 2.5 (4.1%) 12.5 (67.6%) 6 (32.4%) Unsupported ----------------------------------------------------------------------------------------------------------------333 (87.9%) 46 (12.1%) 3 (75%) 1 (25%) 21.5 (87.8%) 3.0 (12.2%) 7.5 (35.7%) 13.5 (64.3%) 132.5 (92.0%) 11.5 (8.0%) 98.5 (90.8%) 10 (9.2%) 57.5 (96.6%) 2 (3.4%) 12.5 (71.4%) 5 (28.6%) 6.7.2 Critical Team Behaviors in Planning A MANOVA was conducted on all effective and ineffective critical team behaviors observed during planning as shown in Table 6.101. The complete MANOVA table is reported in Appendix A.14. 136 Table 6.113 MANOVA to test the affect of all effective and ineffective behaviors during planning Source Project Support Role Project Support * Role **p<0.01 F 9.920 1.264 1.400 P 0.001** 0.295 0.246 In the analysis on all ineffective behaviors, reported in Table 6.114, project support was significant, F(1,30)=6.00, p=0.020. Treatments with manual support had more ineffective behaviors (mean=1.361, sd=1.551) than treatments with automated support (mean=0.389, sd=0.583). Table 6.114 ANOVA for all ineffective behaviors during planning (transformed: Source PS R PS*R S/PS*R Total *p<0.05 ( x + 1) + x ) DF 1 2 2 30 35 SS 5.7672 4.5547 3.1041 28.8372 42.2632 MS 5.7672 2.2773 1.5521 0.9612 F 6.00 2.37 1.61 P 0.020* 0.111 0.216 In the analysis of all effective behaviors (Table 6.115), project support was significant, F(1,30)=10.81, p=0.003. Treatments with manual support had more effective behaviors (mean=9.361, sd=4.193) than treatments with automated support (mean=5.472, sd=2.458). Table 6.115 ANOVA for all effective behaviors observed during planning Source PS R PS*R S/PS*R Total **p<0.01 DF 1 2 2 30 35 SS 136.11 2.04 21.76 377.83 537.75 MS 136.11 1.02 10.88 12.59 F 10.81 0.08 0.86 P 0.003** 0.922 0.432 A MANOVA was conducted on the individual critical team behaviors during planning as shown in Table 6.116. The complete MANOVA table is reported in Appendix A.14. Table 6.116 MANOVA to test the affect of the critical team behaviors on main effects and interactions during planning Source Project Support Role Project Support * Role **p<0.01 F 4.725 1.002 0.875 P 0.001** 0.491 0.520 The data for the ineffective communication behaviors was dichotomized and then analyzed with logistic analysis. As reported in Appendix A.13, there were no significant effects or interactions. There were too few observations to analyze effective communication behaviors. There were no significant differences in the analysis of effective and ineffective cooperation. Refer to Appendix A.13 for the ANOVA summary tables. There were too few observations to analyze ineffective coordination. In the analysis on effective coordination, summarized in Table 6.117, project support was the significant effect, F(1,30)=5.03, 137 p=0.033. Treatments with automated support had significantly fewer effective coordination behaviors (mean=1.806, sd=01.352) that treatments with manual support (mean=3.306, sd=2.630). Table 6.117 ANOVA for effective coordination behaviors during planning (transformed: Source PS R PS*R S/PS*R Total *p<0.05 ( x + 1) + x ) DF 1 2 2 30 35 SS 6.688 0.258 0.324 39.919 47.189 MS 6.688 0.129 0.162 1.331 F 5.03 0.10 0.12 P 0.033* 0.908 0.886 Ineffective acceptance of feedback had one observation, and could not be analyzed. The data in effective acceptance of feedback was dichotomized and then a logistic analysis was conducted. As reported in Appendix A.13, there were no significant effects or interactions. There were too few observations to analyze ineffective giving feedback. Project support was significant in the analysis conducted on effective giving feedback, F(1,30)=6.94, p=0.013 (refer to Table 6.118). Treatments with automated support had significantly fewer effective giving feedback behaviors observed (mean=1.250, sd=1.088) than treatments with manual support (mean=2.250, sd=1.228). Table 6.118 ANOVA for effective giving feedback behaviors observed during planning Source PS R PS*R S/PS*R Total *p<0.05 DF 1 2 2 30 35 SS 9.000 1.292 5.542 38.917 54.750 MS 9.000 0.646 2.771 1.297 F 6.94 0.50 2.14 P 0.013* 0.613 0.136 No ineffective adaptability behaviors were observed. In the analysis on effective adaptability, there were no significant effects as shown in Appendix A.13. There were too few observations in the ineffective team spirit and morale data to analyze. There were no significant differences in the analysis of effective team spirit and morale behaviors (Appendix A.13). 6.7.3 Critical Team Behaviors in the Design Process A MANOVA was conducted on all effective and ineffective behaviors during the design process as shown in Table 6.119. The complete MANOVA table is reported in Appendix A.14. 138 Table 6.119 MANOVA to test the affect of all effective and ineffective behaviors on main effects and interactions Source Project Support Role Design Phase Project Support * Role Project Support * Design Phase Role * Design Phase Project Support * Role * Design Phase **p<0.01 ***p<0.001 F 0.10 4.88 14.43 1.34 0.48 1.31 1.23 P 0.9835 0.0021** <0.0001*** 0.2424 0.8687 0.2465 0.2508 Design phase and role were significant effects in the analysis of all ineffective behaviors, shown in Table 6.120 (design phase: F(2,62)=14.07, p<0.0001; role: F(2,58)=6.52, p=0.0028). Significantly more ineffective behaviors were observed during preliminary and detailed design compared to conceptual design (Table 6.121). The purchaser was observed with significantly more ineffective behaviors than the other two roles as shown in Table 6.122. Table 6.120 Variance analysis for all ineffective behaviors Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*R*PS Residual CD Residual PD Residual DD **p<0.01 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 58 2 4 4 8 20 44 39 0.10 6.52 0.77 0.0118 14.07 0.95 1.51 1.26 0.2690 1.9707 1.0832 0.9038 0.0028** 0.5468 <0.0001*** 0.4418 0.2097 0.2774 Table 6.121 Multiple comparisons of ineffective behaviors in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 1.29 0.9259** Detailed Design 0.91 0.5463** 0.3796 0.36 1.29 Table 6.122 Multiple comparisons of ineffective behaviors based on role mean Designer Manufacturer *p<0.05 **p<0.01 Manufacturer 0.73 0.1759 Purchaser 1.27 0.7130** 0.5370* 0.56 0.73 139 All of the effective behaviors that were observed were compiled and analyzed as shown in Table 6.123. Role, F(2,59)=4.43, p=0.0161, and design phase, F(2,58)=28.28, p<0.0001, were the significant main effects. Significantly more effective behaviors occurred in preliminary and detailed design compared to conceptual design (refer to Table 6.124). From the comparisons between roles, Table 6.125, on average the purchaser had significantly more effective behaviors compared to the other roles. Table 6.123 Variance analysis of all effective team behaviors Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*R*PS Residual CD Residual PD Residual DD *p<0.05 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 59 2 4 4 8 7 41 39 0.07 4.43 2.05 1.7116 28.28 0.42 2.05 1.23 1.3961 15.7843 10.1103 0.9282 0.0161* 0.0986 <0.0001*** 0.7942 0.0977 0.2960 Table 6.124 Multiple comparisons of effective behaviors in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 7.84 3.5370** Detailed Design 6.54 2.2315** 1.3056 4.31 7.84 Table 6.125 Multiple comparisons of effective behaviors based on role mean Designer Manufacturer *p<0.05 **p<0.01 Manufacturer 5.69 0.1759 Purchaser 7.47 1.9537** 1.7778* 5.52 5.69 A MANOVA was conducted on the individual critical team behaviors as shown in Table 6.126. The complete MANOVA table is reported in Appendix A.14. 140 Table 6.126 MANOVA to test the affect of critical team behaviors on main effects and interactions Source Project Support Role Design Phase Project Support * Role Project Support * Design Phase Role * Design Phase Project Support * Role * Design Phase **p<0.01 ***p<0.001 F 1.36 2.12 6.83 0.73 1.61 1.16 0.97 P 0.1577 0.0096** <0.0001*** 0.8970 0.0082** 0.2319 0.5550 In the analysis of ineffective communication behaviors, Table 6.127, the interaction between design phase and project support was significant, F(4,69)=3.95, p=0.0061. During conceptual design, there were significantly more ineffective communication behaviors in treatments without project support compared to treatments with project support as shown in Table 6.128. For preliminary design, treatments without support had significantly fewer ineffective behaviors compared to treatments with automated support. In treatments with automated support, there were significantly more ineffective behaviors in preliminary design than conceptual and detailed design. In treatments without support, significantly more ineffective communication behaviors occurred during conceptual design compared to preliminary and detailed design. Table 6.127 Variance analysis for ineffective communication behaviors observed during design Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*PS*R Residual CD Residual PD Residual DD *p<0.05 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 50 2 4 4 8 35 28 39 0.79 1.34 0.04 0.0240 2.25 3.95 0.61 1.32 0.1851 0.3518 0.1308 0.8659 0.2406 0.9961 0.1141 0.0061** 0.6590 0.2494 141 Table 6.128 Comparisons between design phase and project support for ineffective communication mean CD auto CD manual CD none PD auto PD manual PD none DD auto DD manual **p<0.01 ***p<0.001 0.17 0.08 0.53 0.58 0.39 0.11 0.25 0.14 CD manual 0.08 0.0833 CD none 0.53 0.3611* 0.4444** PD auto 0.58 0.4167* 0.5000** 0.0556 PD manual 0.39 0.2222 0.3056 0.1389 0.1944 PD none 0.11 0.0556 0.0278 0.4167** 0.4722** 0.2778 DD auto 0.25 0.0833 0.1667 0.2778 0.3333* 0.1389 0.1389 DD manual 0.14 0.0278 0.0556 0.3889** 0.4444* 0.2500 0.0278 0.1111 DD none 0.11 0.0556 0.0278 0.4167** 0.4722** 0.2778 0.0000 0.1389 0.0278 0.7 0.6 0.5 Mean No. 0.4 Observations 0.3 0.2 0.1 0 Automated Manual Project Support Figure 6.10 Comparison of ineffective communication for design phase and project support The effective communication behaviors could not be analyzed using variance analysis. The data were dichotomized and then a logistic analysis was conducted. There were no significant effects in the analysis on effective communication as reported in Appendix A.13. Ineffective cooperation behaviors were dichotomized and analyzed using the logistic analysis. As reported in Appendix A.13, there were no significant effects. In the analysis on effective cooperation, summarized in Table 6.129, role and design phase were significant (role: F(2,50)=3.58, p=0.0351; design phase: F(2,60)=10.61, p=0.0001). Significantly more effective cooperation behaviors occurred during conceptual and preliminary design than in detailed design as shown in Table 6.130. The purchaser had significantly more effective cooperation behaviors than the other roles (Table 6.131). None A A AB AC A A A CD PD DD B BC 142 Table 6.129 Variance analysis for effective cooperation behaviors Source Effect DF Variance Component F value Probability Between PS R PS*R s/PS*R Within DP DP*PS DP*R DP*R*PS Residual CD Residual PD Residual DD *p<0.05 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 50 2 4 4 8 22 41 34 2.12 3.58 1.18 0.2411 10.61 0.42 2.06 1.29 0.9386 4.8510 1.8510 0.1309 0.0351* 0.3304 0.0001*** 0.7961 0.0964 0.2626 Table 6.130 Multiple comparisons of effective cooperation in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 2.92 0.3889 Detailed Design 1.62 0.9074** 1.2963** 2.53 2.92 Table 6.131 Multiple comparisons of effective cooperation based on role mean Designer Manufacturer *p<0.05 **p<0.01 Manufacturer 2.06 0.0648 Purchaser 2.87 0.7407* 0.8056** 2.13 2.06 Ineffective coordination behaviors were dichotomized and analyzed using logistic analysis, which resulted in no significant effects (refer to Appendix A.13). Design phase was significant, F(2,58)=33.25, p<0.0001, in the analysis on effective coordination behaviors as reported in Table 6.132. From the multiple comparisons (Table 6.133), significantly fewer effective cooperation behaviors occurred in conceptual design than in preliminary or detailed design. 143 Table 6.132 Variance analysis for effective coordination behaviors observed Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*R*PS Residual CD Residual PD Residual DD ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 50 2 4 4 8 13 36 38 1.69 3.06 1.57 0.5426 33.25 1.75 0.97 1.72 0.6997 2.4525 3.3862 0.1957 0.0559 0.1956 <0.0001*** 0.1504 0.4327 0.1096 Table 6.133 Multiple comparisons of effective coordination in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 2.76 1.7407** Detailed Design 2.46 1.4444** 0.2963 1.02 2.76 There were too few observations to analyze ineffective giving feedback. Design phase was significant, F(2,60)=24.02, p<0.0001 (Table 6.134), in the analysis on effective giving feedback. From the multiple comparisons provided in Table 6.135, significantly fewer effective giving feedback behaviors occurred in conceptual design than in preliminary or detailed design. Table 6.134 Variance analysis for effective giving feedback behaviors observed Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*R*PS Residual CD Residual PD Residual DD ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 54 2 4 4 8 17 36 37 0.00 0.34 1.66 0.1931 24.02 0.33 1.84 1.11 0.3002 0.9952 1.1474 0.9967 0.7117 0.1725 <0.0001*** 0.8600 0.1315 0.3668 144 Table 6.135 Multiple comparisons of effective giving feedback in each phase mean Conceptual Design Preliminary Design *p<0.05 Preliminary Design 1.24 0.7407** Detailed Design 1.48 0.9815** 0.2407 0.50 1.24 There were too few ineffective acceptance of feedback observations to analyze. The data were dichotomized and analyzed using logistic analysis. There were no significant effects as reported in Appendix A.13. There were too few ineffective adaptability observations to analyze. Design phase, project support and role were significant effects in the analysis of effective adaptability behaviors as shown in Table 6.136 (design phase: F(2,94)=15.57, p<0.0001; project support: F(2,57)=3.91, p=0.0257; role: F(2,45)=6.92, p=0.0024). While the interaction had a probability of occurrence that based on the decision level would appear to be significant, the Shapiro-Wilks statistic for normality indicated that the degree of confidence for interactions was a border line decision; therefore the interaction was not considered significant. From the comparisons between design phases (Table 6.137), there were significantly more effective behaviors observed in preliminary and detailed design than in conceptual design. The purchaser exhibited significantly more effective behaviors than the other two positions as reported in Table 6.138. From the comparisons for project support (Table 6.139), there were fewer effective adaptability behaviors observed in treatments with automated support compared to the other treatments. For treatments with manual support and without support, there were more effective adaptability behaviors during preliminary and detailed design compared to conceptual design. In preliminary design, participants without support had significantly more effective adaptability behaviors compared to those with automated support. In detailed design, significantly fewer effective behaviors were observed in treatments with automated support than in the other treatments. Table 6.136 Variance analysis for effective adaptability behaviors Source Effect DF Variance Component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*R*PS Residual D Residual M Residual P *p<0.05 **p<0.01 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 1 2 4 4 8 17 36 37 3.91 6.92 1.12 0.0109 15.57 1.28 2.61 0.71 0.2326 0.1169 0.3680 0.0257* 0.0024** 0.3601 <0.0001*** 0.2819 0.0420 0.6780 145 Table 6.137 Multiple comparisons of effective adaptability behaviors in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 0.41 0.4074** Detailed Design 0.49 0.4907** 0.0833 0.0 0.41 Table 6.138 Multiple comparisons of effective adaptability behaviors based on role mean Designer Manufacturer **p<0.01 Manufacturer 0.18 0.00926 Purchaser 0.54 0.3519** 0.3611** 0.l9 0.18 Table 6.139 Comparisons of effective adaptability behaviors for project support levels mean Automated Manual *p<0.05 **p<0.01 Manual 0.36 0.2222* None 0.40 0.2593** 0.0370 0.14 0.36 No significant effects or interactions were found in the analysis of effective team spirit and moral, as reported in the summary table in Appendix A.13. There were too few ineffective team spirit and morale observations to analyze. 6.7.4 Correlations with Critical Team Behaviors Correlations with Critical Team Behaviors during Planning When considering the correlations between the critical team behaviors and a variety of the performance measures some relationships were found as shown in Table 6.140. Table 6.140 Correlations between various critical team behaviors and dependent performance variables Critical Team Behavior Communication Ineffective Cooperation Ineffective Giving feedback Effective Ineffective Team spirit and morale Ineffective Total effective Total ineffective *p<0.05 **p<0.01 Gantt Score -0.711** 0.595* 0.677* -0.690* -0.666* -0.601* -0.754** The correlations between the NASA TLX and critical team behaviors during planning are shown in Table 6.141. 146 Table 6.141 Correlations between various critical team behaviors and the NASA TLX during planning Critical Team Behavior Adaptability Effective Communication Ineffective Cooperation Ineffective Coordination Ineffective Effective Giving feedback Effective Ineffective Team spirit & morale Ineffective Total effective Total ineffective *p<0.05 **p<0.01 Mental -0.613* -0.616* -0.684* -0.624* -0.077 0.081 -0.584* -0.590* -0.160 -0.790** Temporal 0.265 0.467 0.284 0.352 0.086 0.864** 0.366 0.591* 0.380 0.465 Effort -0.040 0.228 -0.008 -0.048 0.164 0.691* 0.085 0.186 0.410 0.090 Frustration 0.418 0.628* 0.678* 0.835** -0.029 0.424 0.546 0.857** 0.119 0.841** TLX -0.007 0.316 0.146 0.268 0.101 0.856** 0.152 0.514 0.272 0.289 Correlations between critical team behaviors and job satisfaction during planning are shown in Table 6.142. Table 6.142 Correlations between various critical team behaviors and job satisfaction during planning Critical Team Behavior Accepting feedback Effective Cooperation Ineffective Communication Ineffective Coordination Ineffective Team spirit & morale Effective Ineffective Total ineffective *p<0.05 **p<0.01 Comfort -0.222 -0.574 -0.631* -0.676* 0.603* -0.741** -0.719** Challenge -0.333 -0.288 -0.113 -0.219 0.292 -0.086 -0.249 Resources -0.688* 0.047 0.140 0.210 0.403 -0.041 0.164 Job Satisfaction -0.334 -0.507 -0.395 -0.520 0.628* -0.432 -0.543 Several correlations between group workload and the external observer ratings of group workload are reported in Table 6.143. 147 Table 6.143 Correlations between group workload and critical team behaviors during planning Critical Team Behavior Communication Ineffective Cooperation Effective Ineffective Coordination Ineffective Giving feedback Effective Team spirit Ineffective Total effective Total ineffective *p<0.05 **p<0.01 Value of group interaction -0.650* -0.125 -0.784** -0.657* -0.465 -0.760** -0.374 -0.786** Degree cooperation -0.260 -0.061 -0.490 -0.253 -0.691* -0.427 -0.509 -0.383 Difficulty of interaction (external) 0.498 0.651* 0.414 0.363 0.082 0.436 0.172 0.536 Degree cooperation (external) -0.525 -0.178 -0.841** -0.670* 0.046 -0.531 0.060 -0.745** Overall group workload (external) 0.580* 0.637* 0.310 0.418 0.324 0.301 0.480 0.422 Correlations with Critical Team Behaviors during Design Table 6.144 shows the correlations between the NASA TLX and the critical team behaviors during design. Table 6.144 Correlations between critical team behaviors and the NASA TLX during design Critical Team Behavior Accepting feedback Effective Adaptability Effective Communication Effective Ineffective Cooperation Ineffective Coordination Effective Ineffective Giving feedback Effective Ineffective Total effective Total ineffective *p<0.05 **p<0.01 Mental -0.266 -0.341* -0.450** -0.074 -0.197 -0.332* -0.148 -0.224 -0.364** -0.268* -0.19 Temporal 0.236 0.344* -0.174 -0.151 0.313* 0.339* 0.431** 0.190 0.254 0.215 0.308* Effort 0.153 0.260 -0.128 -0.350* 0.022 0.002 0.065 0.247 -0.005 0.138 -0.155 Frustration 0.334* 0.075 -0.057 -0.167 0.253 0.233 0.295* 0.331* 0.388** 0.178 0.250 TLX 0.236 0.183 -0.238 -0.263 0.215 0.242 0.320* 0.320* 0.162 0.200 0.158 The correlations between the critical team behaviors and job satisfaction during design are shown in Table 6.145. 148 Table 6.145 Correlations between various critical team behaviors and job satisfaction Critical Team Behavior Accept Effective Coordination Ineffective Give feedback Ineffective *p<0.05 **p<0.01 Comfort -0.300* -0.312* -0.291* Challenge -0.280* -0.054 -0.096 Resources -0.308* -0.262 -0.322* Job Satisfaction -0.341* -0.241 -0.273* Some of the correlations between group workload and critical team behaviors are shown in Table 6.146. Table 6.146 Correlations between various critical team behaviors and group workload scales Critical Team Behavior Accepting feedback Effective Adapt Effective Cooperation Effective Ineffective Coordination Effective Ineffective Giving feedback Effective Ineffective Team spirit & morale Effective Ineffective Total effective Total ineffective *p<0.05 **p<0.01 Value Difficulty Cooperation Overall workload Value Difficulty (external) Overall workload (external) 0.333* 0.457** -0.179 0.396** 0.364** 0.332* 0.488** 0.329* 0.084 0.303* 0.355** 0.368** -0.013 -0.029 0.413* -0.119 -0.093 -0.114 -0.334* -0.227 -0.082 -0.282* 0.012 -0.232 0.101 0.172 -0.002 0.364** 0.020 0.164 0.131 0.322* -0.060 0.130 0.075 0.253 -0.226 -0.219 0.204 -0.280* -0.217 -0.111 -0.290* -0.471** -0.073 -0.352** -0.169 -0.348* 0.429** 0.289* -0.155 0.279* 0.303* 0.251 0.388** 0.297* -0.041 0.055 0.269* 0.211 0.060 0.042 0.234 -0.077 0.291* -0.077 0.154 -0.096 0.063 -0.066 0.314* -0.103 0.187 0.285* 0.038 0.420** 0.208 0.374* 0.343* 0.397** 0.274* 0.454** 0.321* 0.542** The correlations between various design performance measures and critical team behaviors are shown in Table 6.157. 149 Table 6.147 Correlations between various critical team behaviors and design performance Critical Team Behavior Communication Effective Ineffective Coordination Effective Give feedback Effective Ineffective Team Spirit & Morale Ineffective *p<0.05 **p<0.01 Cost Effective -0.102 0.107 0.267 -0.029 -0.184 -0.576* System Effectiveness 0.255 0.486* 0.717** 0.037 0.194 -0.424 Life-cycle Cost 0.167 -0.107 -0.199 0.048 0.278 0.572* 6.8 Supplemental Group Process Observations Planning A MANOVA was conducted on supplemental group observations during planning (time, money, and non-task related comments) as shown in Table 6.148. The complete MANOVA table is reported in Appendix A.14. Table 6.148 MANOVA to test the affect of supplemental group observations on main effects and interactions during planning Source Project Support Role Project Support * Role *p<0.05 F 4.076 1.595 0.564 P 0.010* 0.149 0.803 In the analysis on time-related comments, project support was significant, F(1,30)=6.27, p=0.018 (Table 6.149). Participants with manual support (mean=5.33, sd=3.511) had significantly more timerelated comments during planning compared to those with automated support (mean=3.0, sd=2.054). Table 6.149 ANOVA for time-related comments during planning Source PS R PS*R S/PS*R Total *p<0.05 DF 1 2 2 30 35 SS 50.174 25.347 15.931 239.958 331.410 MS 50.174 12.674 7.965 7.999 F 6.27 1.58 1.00 P 0.018* 0.222 0.381 In the analysis of money-related comments, shown in Table 6.150, role was significant, F(2,30)=6.58, p=0.004. From the multiple comparisons in Table 6.151, the purchasers made significantly more money-related comments than the designer and manufacturer. 150 Table 6.150 ANOVA for the money-related comments during planning Source PS R PS*R S/PS*R Total **p<0.01 DF 1 2 2 30 35 SS 11.674 49.181 13.181 112.042 186.076 MS 11.674 24.590 6.590 3.735 F 3.13 6.58 1.76 P 0.087 0.004** 0.189 Table 6.151 Multiple comparisons between roles for money-related comments mean 1.8 1.9 Manufacturer 1.9 0.0417 Purchaser 4.3 2.5000** 2.4583** Designer Manufacturer **p<0.01 In the analysis of non-task related comments, summarized in Table 6.152, project support was significant, F(2,30)=9.45, p=0.005. There were significantly more non-task related comments made in treatments with manual support (mean=0.528, sd=0.717) than in treatments with automated support (mean=0, sd=0). Table 6.152 ANOVA for the non-task related comments during planning Source PS R PS*R S/PS*R Total **p<0.01 DF 1 2 2 30 35 SS 2.5069 0.3389 0.3389 7.9583 11.2431 MS 2.5069 0.1944 0.1944 0.2653 F 9.45 0.73 0.73 P 0.005** 0.489 0.489 Design Process A MANOVA was conducted on the supplemental group observations during design as shown in Table 6.153. The complete MANOVA table is reported in Appendix A.14. Table 6.153 MANOVA for supplemental group observations during design Source Project Support Role Design Phase Project Support * Role Project Support * Design Phase Role * Design Phase Project Support * Role * Design Phase *p<0.05 ***p<0.001 F 2.74 2.72 10.13 1.25 1.18 0.92 0.80 P 0.0122* 0.0129* <0.0001*** 0.2511 0.0356* 0.6095 0.0939 Role, design phase, and the interaction between project support and role were significant in the analysis of time-related comments (role: F(2,54)=4.34, p=0.0180; design phase: F(2,57)=9.71, p=p.0002; project support * role: F(2,54)=3.14, p=0.0216) as summarized in Table 6.154. The post hoc comparisons for the significant effects are shown in Table 6.155-Table 6.157. Significantly more time related comments occurred in preliminary design than in the other two design phases. The purchasers made 151 significantly more time-related comments than the others. From the interaction shown in Figure 6.11, in treatments with manual support, purchasers made significantly more time-related comments than the others. Purchasers with manual support made more time related comments than those with automated support and those without support. As reported in Appendix A.3, two variances groupings could have been selected for the analysis of time related comments. Appendix A.15 contains the analysis for variance grouping by role and the results were similar to those of the design phase grouping. Table 6.154 Variance analysis for time-related comments Source Effect DF Variance component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*R*PS Residual CD Residual PD Residual DD *p<0.05 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 18 2 4 4 8 14 32 36 2.64 4.34 3.14 0.4116 9.71 0.13 0.99 1.46 0.3276 1.3534 0.9420 0.0804 0.0180* 0.0216* 0.0002*** 0.9704 0.4200 0.1886 Table 6.155 Multiple comparisons of time-related comments in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 1.40 0.7778** Detailed Design 0.84 0.2222 0.5556** 0.62 1.40 Table 6.156 Multiple comparisons of time-related comments based on role mean Designer Manufacturer *p<0.05 **p<0.01 Manufacturer 0.81 0.1852 Purchaser 1.42 0.7870** 0.6019* 0.63 0.81 152 Table 6.157 Multiple comparisons of time-related comments based on project support and role mean Auto Des Auto Manf Auto Pur Manual Des Manual Manf Manual Pur None Des None Manf *p<0.05 **p<0.01 0.89 1.28 1.31 0.33 0.67 2.36 0.67 0.5 Auto Manf 1.28 0.3889 Auto Pur 1.31 0.4167 0.0278 Manual Des 0.33 0.5556 0.9444 0.9722* Manual Manf 0.67 0.2222 0.6111 0.6389 0.3333 Manual Pur 2.36 1.4722* 1.0833* 1.0556* 2.0278* 1.6944** None Des 0.67 0.2222 0.6111 0.6389 0.3333 0.0000 1.6944** None Manf 0.5 0.3889 0.7778 0.8056 0.1667 0.1667 1.8611* 0.1667 None Pur 0.58 0.3056 0.6944 0.7222 0.2500 0.0833 1.7778* 0.0833 0.0833 2.5 2 1.5 1 AB B AB AB A C Mean No. Observations Designer Manufacturer Purchaser AB 0.5 0 Automated AB AB Manual Project Support None Figure 6.11 Comparisons of time-related comments for the interaction between project support and role Role, design phase, and the interaction between project support and design phase were significant effects in the analysis of money-related comments (role: F(2,54)=7.19, p=0.0017; design phase: F(2,58)=17.33, p<0.0001; project support * design phase: F(4,64)=4.00, p=0.0058), as shown in Table 6.158. Significantly more money-related comments occurred in preliminary design than in the other design phases (Table 6.159). In the comparisons between roles (Table 6.160), purchasers made the most money-related comments. Table 6.161 contains the multiple comparisons for the interaction between project support and design phase. In treatments with automated and manual support, there were significantly more money-related comments in preliminary and detailed design compared to conceptual design as shown in Figure 6.12. For treatments without support, there were significantly more comments in preliminary design than in conceptual and detailed design. In conceptual design, there were fewer money-related comments made in treatments with automated support than in the other treatments. During preliminary design, there were significantly more money-related comments made in treatments without support compared to treatments with automated support. During detailed design, there were significantly more comments in treatments with manual and automated support than in treatments without support. 153 Table 6.158 Variance analysis for money-related comments Source Effect DF Variance component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*R*PS Residual CD Residual PD Residual DD **p<0.01 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 54 2 4 4 8 8 41 37 1.95 7.19 1.31 0.5127 17.33 4.00 1.46 0.50 0.7205 8.0181 3.3580 0.1523 0.0017** 0.2788 <0.0001*** 0.0058** 0.2236 0.8549 Table 6.159 Multiple comparisons of money-related comments in each phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 3.41 2.3333** Detailed Design 1.59 0.5185 1.8148** 1.07 3.41 Table 6.160 Multiple comparisons of money-related comments based on role mean Designer Manufacturer **p<0.01 Manufacturer 1.56 0.0556 Purchaser 3.02 1.5185** 1.4630** 1.5 1.56 Table 6.161 Comparisons of money-related comments based on project support and phase mean CD auto CD manual CD none PD auto PD manual PD none DD auto DD manual *p<0.05 **p<0.01 0.47 1.39 1.36 2.25 3.56 4.42 1.92 2.36 CD manual 1.39 0.9167* CD none 1.36 0.8889* 0.0278 PD auto 2.25 1.7778* 0.8611 0.8889 PD manual 3.56 3.0833** 2.1667** 2.1944** 1.3056 PD none 4.42 3.9444** 3.0278** 3.0556** 2.1667* 0.8611 DD auto 1.92 1.4444** 0.5278 0.5556 0.3333 1.6389 2.5000** DD manual 2.36 1.8889** 0.9722* 1.0000 0.1111 1.1944 2.0556* 0.4444 DD none 0.50 0.0278 0.8889 0.8611 1.7500* 3.0556** 3.9167** 1.4167* 1.8611** 154 5 4 3 Mean No. Comments 2 1 0 D CD BC BC BF A Automated CE BEF AF CD PD DD Manual None Project Support Figure 6.12 Comparing the money-related comments for project support and design phase In the analysis on non-task related comments (Table 6.162), design phase and the interaction between design phase and project support were significant (design phase: F(2,56)=8.44, p=0.0006; design phase*project support: F(4,63)=3.72, p=0.0089). As shown in Table 6.163, there were significantly more non-task-related comments observed in detailed design than in conceptual and preliminary design. From the analysis on the interaction (Table 6.164 and Figure 6.13), in detailed design, significantly more nontask related comments occurred in treatments with automated support than in the other treatments. Table 6.162 Variance analysis for non-task-related comments Source Effect DF Variance component F value Probability Between PS R PS*R S/PS*R Within DP DP*PS DP*R DP*R*PS Residual CD Residual PD Residual DD **p<0.01 ***p<0.001 Fixed Fixed Fixed Random Fixed Fixed Fixed Fixed Random Random Random 2 2 4 53 2 4 4 8 17 25 39 0.21 1.02 0.87 0.5127 8.44 3.72 1.35 1.31 0.1003 0.1408 0.5544 0.8081 0.3689 0.4866 0.0006*** 0.0089** 0.2620 0.2548 Table 6.163 Multiple comparisons of non-task-related comments in each design phase mean Conceptual Design Preliminary Design **p<0.01 Preliminary Design 0.2 0.0926 Detailed Design 0.5 0.4537** 0.3611** 0.09 0.2 155 Table 6.164 Comparisons of non-task related comments by phase and project support mean CD auto CD manual CD none PD auto PD manual PD none DD auto DD manual *p<0.05 **p<0.01 1 CD manual 0.2 0.1667 CD none 0.0 0.0556 0.2222 PD auto 0.06 0.0000 0.1667 0.0556 PD manual 0.1 0.0833 0.0833 0.1389 0.0833 PD none 0.4 0.3056 0.1389 0.3611 0.3056 0.2222 DD auto 0.9 0.8056** 0.6389** 0.8611** 0.8056** 0.7222** 0.5000* DD manual 0.4 0.3889 0.2222 0.4444* 0.3889 0.3056 0.0833 0.4167 DD none 0.3 0.2778 0.1111 0.3333 0.2778 0.1944 0.0278 0.5278 0.1111 0.06 0.2 0.0 0.06 0.1 0.4 0.9 0.4 C Mean No. 0.5 Comments BC CD AB ABC PD DD AB AB AB 0 AB A Automated Manual None Project Support Figure 6.13 Comparing non-task related comments for project support and design phase 6.9 Correlations There were positive relationships between cost effectiveness and doubt ranging from a low of 0.329 to a high of 0.468 as shown in Table 6.165. System effectiveness had a positive relationship with doubt. A significant negative relationship existed between material cost and doubt. 156 Table 6.165 Correlation between doubt and various measures of design performance Doubt Conceptual design 0.329* -0.068 0.333* -0.295 0.330* 0.229 -0.040 Doubt Preliminary design 0.472** -0.362* 0.069 -0.434** 0.409* 0.120 0.110 Doubt Detailed design 0.468** -0.310 0.149 -0.443** 0.220 0.083 0.012 Reflective Best of ability 0.371* -0.310 0.165 -0.438** 0.507** 0.421* 0.388* Reflective Liked system 0.230 -0.220 0.190 -0.372* 0.452** 0.416* 0.327 Cost effective Life-cycle cost Design cost Material cost System effectiveness Reliability Robustness *p<0.05 **p<0.01 Correlations between perceptions of time and temporal ratings and number of time-related concepts and temporal demand are shown in Table 6.166. Table 6.166 Correlation between time-comments, perceptions of time and temporal demand Conceptual design Preliminary design Detailed design *p<0.05 Correlation between temporal demand and time-related comments 0.326 0.461 -0.203 Correlation between temporal demand and time perceptions -0.071 -0.526* -0.090 The correlation between the difficulty of interacting and the perceptions of group member competence and helpfulness are reported in to Table 6.167. All correlations were significant and negative. Table 6.167 Correlation between difficulty of group interaction and perceptions of competence and helpfulness Correlation between difficulty and competence -0.459* -0.473* -0.476* Correlation between difficulty and helpfulness -0.596* -0.460* -0.411* Conceptual Design Preliminary Design Detailed Design *p<0.05 Correlations between the TLX and the overall group workload are shown in Table 6.168. Only the correlation between TLX and overall group workload during planning was significant. Table 6.168 Correlation between average TLX and overall group workload Variable Planning Conceptual Design Preliminary Design Detailed Design Reflective *p<0.05 Pearson Correlation 0.582* 0.164 0.350 0.136 0.286 157 Chapter 7 Discussion The data were collected to answer the question, how does varying team design and project support affect performance? This gave rise to three specific questions concerning the performance and the impact on the designers: How was performance affected by team design and project support during the design projects life-cycle? How was mental workload affected by team design and project support during the design projects life-cycle? How was job satisfaction affected by team design and project support during the design projects life-cycle? The research involved studying three-factors. The between subjects factors included team design, with two levels (individual and groups of three) and project support, with three levels (no project planning and tracking tools, manual planning and tracking tools, and automated planning and tracking tools). The within subjects factor was design phase which consisted of planning (conditions with project support only), conceptual design, preliminary design, and detailed design. Subjects were randomly assigned to conditions. The data were analyzed in Chapters 4 though 6. There were many significant effects and interactions. Because of the number of analysis of variances tests that were conducted, it is expected that some of the results may be significant due to chance. In an attempt to protect against this, MANOVAs were conducted. However, the MANOVA has the same assumptions as ANOVA for normality and homogeneity of variance, which some of the data sets violated. Some results were easily attributed to aspects of the design process and the nature of the task. Others have support from previous research and some findings raise new questions which will be discussed for future research. Recall that in the data analysis between groups and individuals, an average of the three individual scores was used for the group data point. Refer to Appendix A.17 to see the analysis of the reliability between the group members evaluation. In some cases there was little agreement, further justifying the analysis by role to understand the differences within groups. Also included in Appendix A.17 are the reliability scores between external observers for group workload and critical team behaviors and a brief discussion on the nature of the agreement. 7.1 Performance For this study, two major aspects of performance were captured: design performance and planning/tracking performance. Overall design performance of the deliverable was measured using cost effectiveness which incorporated life-cycle cost with system effectiveness. Planning performance was initially defined using a scale of one to five to determine how complete were the scoping documents and Gantt charts. To determine how well the projects remained on budget and on schedule, the cost performance index (CPI) and schedule performance index (SPI) were calculated at 30 minute intervals throughout the design process during status report meetings. The concept of time was important and was recorded for planning, each design phase, and for each status report meeting. In both engineering and project management literature, reducing cycle time was a frequently reported goal (e.g., Gido et al., 1999 and Adachi, Shih, & Enkawa, 1994). 7.1.1 Design Performance How was performance affected by team design and project support during the design projects life-cycle? 158 Hypothesis 1: Treatments that used project support were expected to have more cost effective designs than those in the control (no project support). Groups were also expected to have more cost effective designs than individuals. This hypothesis was not supported. Cost effectiveness was the primary variable used to evaluate the performance of each design. Cost effectiveness was the ratio between system effectiveness and lifecycle cost. Because cost effectiveness is a ratio, the components used to compute the ratio may have had significant effects that were masked. Therefore, each of the components used to calculate the cost effectiveness ratio was analyzed as supplemental performance indicators. The numerator of cost effectiveness was defined by system effectiveness, which was an average of the reliability and robustness scores added to the producibility score (a score based on the time to build the system), and size score. While system effectiveness did not have significant effects, team design was a significant effect in the reliability analysis. Groups had significantly higher reliability scores compared to individuals. Groups consistently had systems that were able to at least cross the barrier, if not the finish line. Only 11% of the groups designed systems that completely failed compared to 44% of the systems designed by individuals. Overall the reliability of systems created by groups was 98% higher than that of systems created by individuals. This is evidence that individuals had too many details to attend during the design process and system reliability was sacrificed (Svensson et al., 1993; Gaillard, 1993; Jorna, 1992; Eggemeier, 1988; Gopher et al., 1986). In the comparison between groups, there were sufficient resources to deal with the various design requirements. These results in this study were more dramatic than in an earlier study that compared groups of three with groups of six; smaller groups designed systems that were six percent more reliable than systems designed by the larger groups (Meredith, 1997). The main difference between these two studies was that the impact between individuals and groups was more dramatic than the difference between different group sizes. One hundred percent of the groups with automated support compared to 33% of individuals with automated support were able to get the ball past the barrier. Similarly more groups with manual support (83%) were able to cross the barrier compared to individuals (50%). However, 83% of both individuals and groups without project support designed systems capable of crossing the barrier. For individuals, automating the project support is a distraction caused by a lack of resources (Hockey et al., 1989). Group members provided extra attentional resources which compensated for the extra workload (Davis et al., 1996). In the treatments with manual support, percentages of success began to even out between groups and individuals, indicating that the manual project support was a greater source of distraction for groups but less of a distraction for individuals compared to the automated support. Finally when all attention was focused on the designing without project support, the performance was identical between groups and individuals based on the criteria of crossing the barrier. Robustness scores were low in general. An interesting trend (but not significant) was groups with automated support had slightly higher scores than the other groups. The opposite trend occurred for individuals. None of the individuals with automated support successfully completed the robustness test. That none of the individuals with automated support were able to past the robustness test was expected since only 33% passed the reliability test. Only those designs able to have the payload cross the barrier in the reliability test were able to attempt the robustness test. The robustness results further support the claim that the individuals were operating without extra resources to spare and made a tradeoff between completing the design on time and designing the system to meet all system requirements (Rolfe et al., 1973). The robustness scores for individuals with automated support may indicate an overload. Previous research had found that individuals dealt with increased workloads in one of two ways: either they found more efficient ways to complete tasks or they omitted sub-tasks (Rolfe et al., 1973). Regardless of the coping strategy the result was degraded performance. In the design literature this was referred to as satisficing, which is when designers looked for a single satisfying solution as opposed to thoroughly 159 comparing all alternatives (Baker et al., 1991). In this case, the individuals may have decided not to worry about designing for robustness in lieu of completing a testable design. Life-cycle costs were in the denominator. Prior research found that design costs in groups of six were 41% higher than the design costs in groups of three, which when considering the costs due to labor was expected (Meredith, 1997). Translating the results to this study, as expected the life-cycle costs were higher for groups (by 26%) than for individuals. Because the major difference is the factor of the group size, it is logical that the percent differences are smaller between one and three members versus three and six members. The materials cost was not significantly different between individuals and groups as expected because they had to operate within the same budget. To better understand why there were no significant effects based on project support, design costs was further explored. Team design and project support were both significant effects in the analysis on design cost. Groups mean design costs were about three times higher than individuals mean design costs, which was logical because there was three times the amount of labor for groups compared to individuals. The design costs were significantly lower in treatments with automated support compared to treatments without project support. The design costs combined with the slight differences in materials costs help explain why lifecycle costs were not significantly different based on project support. However, the finding for design cost led to questioning how time records were kept in groups. In theory, all individuals had up to $120 to spend on labor (at $1/minute up to two hours). As can be seen in the average cost, individuals averaged slightly over one and one-half hours on the design process. Note participants were instructed that the two hour period included both the design, building and testing of the system, but building and testing was not included in the design time. Groups had a mean design cost of $280. The average time groups spent on design was only two minutes longer than the individuals average. However, some of the groups with automated support were able to design in a savings from labor and reallocate that money to materials, hence their higher average material costs without exceeding the cost constraint. What was different was how the various groups accounted for their billable time. All groups with manual support or without project support made the assumption that all three group members worked the same amount of time throughout the design process. Therefore these groups multiplied their overall design time by three to determine the labor hours. This method for determining design cost was used regardless of if group members were idle or actively engaged in the design process. However, in groups with automated support, half of the groups (three) tracked their individual labor hours to determine the pro-rated billable time, which introduced a cost savings. In subsequent design phases (manufacturing and testing), more groups recognized they should only charge the labor hours for group members participating in the activities, which resulted in a slight cost savings. Note the manufacturing and testing activities called for one builder and one observer. As long as the third person did not participate they were allowed to do this to just charge for the time of two as opposed to three participants. In the latter scenario, two groups without project support, one group with manual support and four groups with automated support took advantage of this cost savings. In general, groups with automated support were more sensitive to time and cost issues which will be addressed in more detail in the group process discussion. The average cost for groups was less than the cost constraint while for individuals the average exceeded the cost constraint. In actuality, 33% of groups with automated support, 50% of groups with manual support, and 50% of groups without project support exceeded the cost constraint, whereas 67% of individuals with automated support, 83% of individuals with manual support, and 33% of individuals without support exceeded the cost constraint. This result was an indication that the task demands in treatments with project planning and tracking were too great for a single individual to handle. Embedded in the life-cycle cost was the number of moving parts and total parts. These were factored in as contributing to the maintenance costs. Individuals had significantly more moving parts than groups which helped explain why the individuals average life-cycle cost was not lower. This again 160 provides another example of a design requirement that was omitted by most individuals in order to complete the project on time. The result that design costs were lower in treatments with automated support compared to treatments without project support suggested that participants with automated support could attend to the multiple expected outcomes better than those without planning support and those with manual support. The computer seemed to help create an awareness of the relationship between key variables (in this case time and money) that was lacking in the other treatments. In addition the computer helped to organize a large amount of information more concisely than was possible in the other treatments. This result differed from other research on automation where large volumes of information created by the computer degraded performance (Hockey et al., 1989; Salvendy, 1981). Because group participants with automated support identified the relationship between labor costs and design costs, they even exercised the notion of billable hours, a concept not addressed during training, versus using the assumption that everyone worked equal amounts of time as previously discussed. Previous research found that implicit and explicit communication determined effective team functioning (Volpe, Cannon-Bowers, Salas, & Spector, 1996; Kleinman, 1990). As part of the group process analysis, observations were recorded on the number of time and money-related comments that groups made during design. Design role, design phase, and the interaction between project support and role were significant effects in the analysis of time-related comments during the design process. The aspects that were relevant to performance were the differences in the interaction between project support and role. Purchasers seemed to be at the root of making time-related comments as indicated by the difference based on role alone. And when adding the project support into the picture, purchasers with manual support made more time-related comments than purchasers with automated support or those without support. The result in groups without support indicated that those groups were not as concerned with time and therefore labor cost. On the other hand, because many of groups with automated support tracked their time, the fewer time related comments might be an indication that there was implicit communication occurring where the concern for time was left unverbalized. Groups with manual support were more explicit because their coordinating mechanisms were not as formalized, or organized for that matter, as the automated groups. This was supported by the observation that on average, purchasers made twice the number of time-related comments with manual support compared to the other groups. These findings provided some support that the concept of time was implicit in the groups with automated support. Planning prior to the start of design may have created an awareness, which carried over into tracking such that these groups did not need to expend extra effort by discussing time or mentioning the time that remained because of their heightened awareness due to the tracking mechanisms that were in place. Groups with manual support may have suffered from information overload or poor tracking methods. One of the reasons behind automating a function is to reduce fatigue (Harris et al. 1995). Many of Gantt charts developed manually were difficult to read because of small and illegible writing and erasures and scratch outs. Too much information was placed on one sheet of paper and time scales were not accurately drawn which also made interpreting the chart and quickly showing progress difficult. Note that for most of the charts, the first hour of the schedule tended to be neat but the second hour of activities were difficult to follow. The decrease in legibility may indicate the onset of fatigue. Participants with manual support could not start the Gantt chart until the work breakdown structure and resource assignments were started and most waited until these activities were complete. Therefore the combination of time pressure and rushing to complete the chart could result in fatigue especially if the activity was completed by a single person. For groups without support, time had not been previously discussed and the only way to address it was to have the purchaser, who was in charge of the time log, serve as the reminder for the group. The significant interaction between project support and design phase from the analysis of moneyrelated comments is also relevant to this discussion. In conceptual design, there were fewer moneyrelated comments made by groups with automated support (about one for every two groups) compared to 161 groups with manual support and groups without support (approximately one per group). In preliminary design, there were significantly more money-related comments made by groups without support (about two more per group than automated) compared to groups with automated support. In detailed design there were significantly more comments made by groups with manual and automated support than groups without support. For money-related comments made in conceptual and preliminary design, the findings indicated groups with automated support implicitly understood the importance of cost in the final performance evaluation. They did not need to directly address money as evidenced by the fewer comments made by groups with automated support compared to the other groups. However, during detailed design communication on this topic became explicit for groups with project support compared groups without support. This may be explained by last minute discussions on ways to save costs on materials. Anecdotally, several groups in the last phase used leftover time to look for ways to reduce costs. These results suggested that the performance of participants with automated support (based on reliability and robustness) may have been too concerned about completing the project according to their schedule because there was a hint that the performance was slightly lower in this scenario, particularly with the individuals. After considering the number of individuals that exceeded the cost constraints with project support, the individuals in general seemed to be overloaded based on the task demands (note that this will be explored further in the NASA TLX and job satisfaction discussions). Training or other avenues to ensure individuals and groups do not sacrifice quality for time savings should be explored more thoroughly especially considering that none of the individuals or groups exceeded the two hour limit set for design. The task for this study could be classified as a multistage and conjunctive task. In other words, success depends on multiple group members as opposed to one member. Performance was found to be better in groups than individuals in conjunctive, multistage tasks (Smith, 1989). In Smiths study, groups of three performed 50% better on three-stage tasks and 51% better on conjunctive tasks than nominal groups. After considering performance from multiple perspectives, the results from this study provided mixed support for this claim, with reliability and moving parts the only aspects supporting the finding based on statistically significant differences. Recall groups performed 97% better than individuals based on reliability and 48% better based on the number of moving parts. While the content of the task in Smiths study differed from the design task in this study, groups performed significantly better than the individuals on the overall work product in both studies. While groups performed better than individuals from several perspectives, the cost associated with groups must be a strong consideration in the decision to form groups, if life-cycle cost is selected as the primary variable of interest. Although from the consideration of being remain within cost constraints, groups appeared to do a better job than individuals. Even though the reliability score for groups was almost twice as high as the reliability score for individuals, the labor cost moderated these results. It was surprising though that groups with a design cost almost three times as high as that of individuals and reliabilities almost twice as high averaged out the differences to result in similar cost effectiveness ratios. In other words, it would appear that the cost savings might justify the lack of reliability in the design; or in cases where reliability is important the added cost is justified. Understanding the design goal is critical prior to determining if groups or individuals should be used. Of course, caution must be used in interpreting these results because the participants were forced to complete the project in two hours. As long as detailed design was completed within the time limit (all finished within the two-hour mark) participants were allowed to finish the building and the testing, recognizing that the excess time was factored into life-cycle cost. Four of the six individuals without project support exceeded the two hour limit and one finished 30 minutes early (including building and testing). Two of the six individuals with manual support exceeded the time limit, while only one individual with automated support exceeded the time limit. Two of the groups without project support exceeded the time limit, and two each of the supported teams exceeded the time limit. 162 The design performance results supported the idea that the design task represented a complex set of problems and decisions. There were several layers of information that were presented: the planning objectives, the design specifications and deliverables, and the objectives for each role. Individuals received all of the information. In the case of the groups, each person was provided all of the planning and task information, but the objectives for a given role were provided to a single individual; some of whom overtly shared the information, some of whom chose not to share the information. The variables that were used to calculate cost effectiveness in addition to several other variables were developed as measures of how well the role objectives were met. The only overt measure used in the cost calculation was the number of moving parts which was rolled into the calculation of maintenance costs (more subtle was the total number of parts and number of unique parts). In the analysis of the number of moving parts team design was significant. This may be an indication that individuals may have had to satisfice to finish their task as opposed to considering ways to optimize the various aspects of the design. This concept can be further researched since individuals were not asked to disclose their thought process as they worked through the design process. Groups often directly shared their thoughts and opinions about how to improve their designs and create cost savings whether it was through material substitution, reducing the systems size, or keeping the design simple, such that the manufacturing time would be minimal and the number of parts/moving parts would be as low as possible. Because the participants had varied experiences and backgrounds, correlations were conducted between the demographic information and performance as shown in Appendix A.18. Few strong correlations existed. The strongest was between the number of errors and number of previous design projects the participant worked on (r=0.710), suggesting that as design experience increased, the number of errors that were made increased. This may have been due to the complexity of the designs developed by those who had a lot of experience. In this design project, a simple design translated into a design that was well aligned with the evaluation criteria in terms of ensuring the design requirements were met within the time and cost constraints. Supplemental Design Question Supplemental questions were asked after each design phase and at the end requiring reflection over the entire design process. After each phase was complete, participants were questioned about their doubt that they could accomplish the task. In the analysis over time, design phase was a significant effect; the mean score was significantly higher during conceptual design compared to detailed design. Initially, the participants had very little doubt in their ability to create a system to meet the design requirements. The mean score of 5.4 [out of 7] indicated that they tended to agree/slightly agree that there was no doubt they could develop a system. However, by the end of detailed design, they were neutral on their ability. Supplemental Design Question Reanalyzed based on the Role in Groups When considering the individual responses within the groups, there were two significant effects in the analysis of doubt during design. Participants with project support tended to be more doubtful of their work compared to those without support, which was also reflected in the comparison between groups and individual. The lack of confidence in groups with project support might be due to the ability of the participants to project their results into the future and see weaknesses due to their awareness of the steps ahead. Was this a good thing? Maybe not, but it raised an awareness amongst the participants that might be useful in real situations where extensions could be secured. The tight time frame left little flexibility for participants to thoroughly explore ideas. After the initial design phase, the participants had a decrease in their belief that they could successfully complete the project. This finding was also logical. In fact one of the justifications used to support using a project management approach is an increased awareness of the resources and schedule (Gido et al., 1999). This result supported the concept that at the beginning of projects, participants all 163 believe they have the ability to complete the project within the appropriate time frame and using the appropriate resources (Weiss and Wysocki, 1992). And even after conceptual design there was a fairly strong belief that one of the ideas generated would in fact meet the expectations specified in the design problem. However, as time elapsed and tradeoffs were made, doubt increased. As an awareness of time running out became more evident, the doubt in ability persisted. A promising observation was that doubt did not change between preliminary and detailed design. Once the seed of doubt was planted it remained, and those who were confident at the end of preliminary design remained confident during detailed design. 7.1.2 Design Cycle Time Teams were expected to take longer to complete the design project than individuals. Treatments with manual project support were expected to take longer to complete compared to the other treatments. Hypothesis 2: This hypothesis had mixed support. The overall design cycle time for individuals was slightly shorter (not significantly) than the design cycle time for groups. This result was surprising because of the large literature base indicating groups tended to take more time than individuals to complete a task. The reason for the lack of a difference was that towards the end of the design process groups could approach tasks by dividing the work amongst members. When one group member finished a task early, he or she could help finish other uncompleted tasks if needed. In early design activities, all members discussed and debated issues, such as the best system to design. Groups had to deal with multiple decision makers. Individuals saved time because they did not have to persuade others to accept their decision. Considering the overall design time, treatments without support had significantly higher design cycle times (by 17%) compared to treatments with automated project support. While the cycle time for treatments with manual support was slightly longer than that of automated support and slightly shorter than that without project support, these differences were not significant. When considering the time in each design phase, design phase and the interaction between design phase and team design were significant. Conceptual design was the shortest design phase (averaged 12 minutes). There was little difference between preliminary (averaged 42 minutes) and detailed design (averaged 41 minutes). From a task-related perspective this result made sense because the conceptual design consisted of identifying design criteria and brainstorming concepts. The design phase ended when no new design concepts could be identified. Almost equivalent amounts of time were spent determining which concept to select for detailed design and converting the concept into detailed drawings and manufacturing instructions. This was because the participants were required to thoroughly consider two concepts prior to selecting one for detailed design. This requirement was based on prior research indicating that higher performing designers developed and explored more alternatives compared to the lower performing designers (Ehrlenspiel et al., 1993). As reflected in the interaction, the mean time in each phase was shorter during conceptual design than in the other design phases for both groups and individuals. However, groups took significantly longer during preliminary design than in detailed design, and detailed design was longer than conceptual design. Other than conceptual design being the shortest phase, there were no other significant differences for individuals. Individuals took significantly longer during detailed design than groups. This result explains why the individual and group differences were not significant for the overall design cycle time. In addition, this result supports the idea of approaching the task using a division of labor. Previous research supports that when a division of labor can be used groups are superior to individuals (Hare, 1976). While the division of labor was used, the groups still collaborated. In other words, even though participants worked on different activities, they did not work in isolation. 164 According to the plan provided in the groups Gantt charts, two-thirds had all members contributing to all aspects of this phase. This was acknowledged in the labor time recorded for groups where all members worked for the entire conceptual design phase. As mentioned previously, during preliminary design, the groups began to approach some aspects of the phase individually and some aspects collaboratively. For example, determining which two concepts to explore, determining the final design for embodiment, and reviewing the output produced at the end of the phase to ensure the final products were complete tended to involve all group members. However, many of the activities after these decisions were made were approached through a division of labor. In detailed design, a true division of labor existed. The designer was the primary creator of all detailed drawings. The manufacturer predominately completed the manufacturing instructions. The purchaser was responsible for the concept selection justification, bill of materials and life-cycle costs. There was no consistency in the way participants completed their responsibility and helped others. However, individuals had to complete all aspects of the task. Therefore, they were able to save time during some activities (for example, in decision making, they did not have to convince someone to their way of thinking). 7.1.3 Planning Performance Cost variance and schedule variance were expected to be inferior for treatments with manual project support compared to those with automated support. Cost variance and schedule variance were expected to be inferior for individuals compared to groups. The planning time was expected to be greater for groups compared to individuals. Planning time was anticipated to be greater for treatments with manual project support compared to automated project support. Hypothesis 3: Cost Performance Index and Schedule Performance Index This hypothesis received mixed support. The primary measures of performance were how well the participants were able to remain on schedule and within their budget using the schedule performance index (SPI) and cost performance index (CPI). The indices were selected due to the ease with which the value is interpreted. Ratios with a value >1.0 indicated the performance was progressing faster than planned or spending was under that budgeted. Values =1.0 indicated that the performance was as planned and values <1.0 indicated there was overspending or the project was behind schedule. It was possible to have one of the indices over one and another under one. The limitation with the indices was projects could vary over time from the schedule but as long as compensations were made, the project could be brought back on track without understanding that it was ever off track. However, for convenience this method was selected. There was no evidence to support the hypothesis that the cost variance and schedule variance was inferior for individuals (SPI: (mean=1.15, sd=0.269; CPI: mean=1.12, sd=0.275) than for groups (SPI: mean=0.99, sd=0.207; CPI: mean=0.97, sd=0.219). Similarly, the SPI and CPI for the different types of support were almost identical (SPI automated: mean=1.07, sd=0.275; SPI manual: mean=1.06, sd=0.231; CPI automated: mean=1.07, sd=0.293; CPI manual: mean=1.07, sd=0.234), thus there was no support for the hypothesis that the type project support would impact the indices. The reporting time period was a significant effect for the SPI. Over time, the SPI decreased. At the first status report meeting, the SPI indicated all participants tended to be ahead of schedule. By the second meeting, the participants were just slightly behind schedule. By the third report meeting, participants were more noticeably behind schedule. Like the SPI, over time the CPI became significantly smaller. At the first status report meeting, the CPI indicated that all participants tended to be ahead of schedule in terms of the cost (or under spending). By the second meeting, the participants were just slightly overspending, but in general this was not compensated and by the third meeting the participants were overspending more noticeably, although the averages were close to one. This result was not 165 surprising since most individuals and groups allocated more time than necessary for conceptual design activities. Gantt Chart and Scoping Document Scores A general scoring system was developed to determine the quality of the scoping document and Gantt chart. There was no difference in the scoring of the scoping documents based on team design (groups: mean=4.08, sd=0.9; individuals: mean=4.08, sd=1.0) or project support (automated: mean=4.25, sd=0.75; manual: mean=3.92, sd=1.08). The Gantt chart had significant differences based on the type of support: the charts developed using the software (mean=4.92, sd=0.29) were scored significantly higher than the charts developed manually (mean=2.92, sd=1.24). This result was not surprising. The software was straight forward and multiple iterations on a Gantt chart (in terms of rearranging time and resource allocations) could be easily conducted in a short time period. In addition, the feedback mechanisms in the software were easily accessed and obviously suggested problems in the Gantt chart. For example, if a person was required to work on two tasks simultaneously, their information would appear in red on the resource usage sheet. In treatments with manual support, the participants had difficulty correcting their mistakes and seeing potential scheduling conflicts. When problems were noticed in the Gantt chart, often those using manual support were satisfied with discussing what the Gantt chart should represent without trying to actually fix the chart. The concern with this approach was that tracking of progress in the status report meetings would be more difficult and tedious. Amazingly, the measures that might indicate this (status report time, CPI, and SPI) did not have significant effects for type of project support. Supplemental Planning Performance Questions To help understand the participants perceptions regarding the plans they developed, several supplemental questions were asked including doubt about ability to complete the design project according to the plan and within the budget, developing the best plan possible and questions regarding the specific tools they had used for the planning process. The interaction between team design and project support in the analysis of doubt in their ability to meet their developed plan was significant. Groups using manual support had more doubt than individuals using manual support. Within groups, there was less doubt for those using automated support than those using manual support. From this result for manual support, individuals were much more confident in their ability to carry out their plan than groups, which makes sense because individuals did not need to generate buy-in to the plan for ownership. Since individuals created their own plan, it was developed with their own abilities in mind. In the groups, the members were novice to working with each other and often the plan was a compromise. Confidence was higher in groups with automated support compared with manual support. This too was logical. With automated support, the tasks, dependencies, time allotment and accountability were clearly laid out with mechanisms in place through the software to reduce the number of potential errors. Some errors that did occur were planning for longer than the two hour time allotment, double scheduling one person for two activities at the same point in time, or unfairly assigning workload all to one person. With manual support, verifying accuracy and correcting errors was more difficult if errors could in fact even be detected. No differences were found for creating the best plan possible, ease of use, efficiency, effectiveness, productivity, and satisfaction during planning. Efficiency, effectiveness, productivity, and satisfaction were not significant in the evaluation of planning and tracking tools used during design. Design phase was significant in the ease of use, which was significantly higher during conceptual design than in either preliminary or detailed design. These differences should not be considered in the design implications because participants did not use the planning and tracking tools until preliminary design. In the reflective evaluation, there were no differences between the types of project support. 166 Supplemental Planning Questions Reanalyzed based on the Role in Groups When considering the results in groups based on the individual roles there were many significant differences in the supplemental planning questions. Groups with automated support were significantly more confident in their ability to carry out their plan compared to groups with manual support. Although both scores tended to be low: groups with automated support were neutral while groups with manual support slightly disagreed that they could complete the design process according to the plan. A similar pattern was observed in the analysis on participants developing the best plan within their ability. These differences may reflect the naivety of the participants with regard to the planning process. Planning had to be completed within 55 minutes. With automated support, there was time for reviewing and iterating through planning steps. However with manual support, developing the Gantt chart was time consuming. To complete the plan in the allotted time, the participants could only develop one complete draft of the chart. After the chart was complete they could immediately see time allocation problems. But to fix the problems, they had to erase sections or write notes on the chart without developing a new version. The difficulty in using manual support was reflected in the answers to supplemental questions. Group members with automated support rated their planning tools higher in term of ease of use, efficiency, productivity, and satisfaction than groups with manual support. During design, the manufacturer rated the planning and tracking tools as easier to use than the designer and purchaser. This was probably because the manufacturer typically did not entirely engage in the status report meetings other than to verbalize his or her progress. Anecdotally, most often the manufacturer was busy with prototyping and let the designer or purchaser lead the meetings. It was surprising that the means were similar between purchasers and designers because purchasers more often led the status report meetings. Not surprisingly, productivity was rated higher in groups with automated support compared to groups with manual support. This may haven been due to the ease with which progress could be tracked using the automated support. In the situation where adjustments were needed the changes could be made with the trickle down effect of changing the entire Gantt chart seemingly instantaneously. From the reflective evaluation, the ease of use, productivity and satisfaction were all rated higher for groups with automated support compared to groups with manual support. These differences did not all emerge during the design process. Only after considering the process from start to finish were the differences between project support levels significantly different (although these were similar to the results found during planning). Time and Money-Related Comments As discussed in the design performance section, looking at the number of time and money-related comments during planning gives insight into implicit and explicit behaviors. In the analysis of timerelated comments, project support was significant. Groups with manual support were observed more frequently making time-related comments during planning compared to those using automated support. This supported the idea that time was implicit in the activities of the participants with automated support while those with manual support needed to elicit more information from each other. In the analysis of money-related comments, role was significant. The purchasers made significantly more money-related comments than the designer and manufacturer. This is not surprising given that the purchasers primary responsibility was to minimize costs and to track the budget. However, the lack of differences between project support types suggested that groups handled the type of communication about money in a similar manner. 167 7.1.4 Planning and Tracking Time The time spent in planning was another indicator of performance. Groups took significantly more time to plan their design project than individuals. Thus the hypothesis that planning would take longer for groups than for individuals was supported, though not surprising. There is a large literature base that supported the finding that groups frequently take more time to complete tasks than individuals (for example, Gustafson et al., 1973; Barker et al., 1991; Hare, 1992). Participants with automated support took significantly less time to complete the planning process compared to participants with manual tools. Therefore there was support for the hypothesis that the planning time would be greater for treatments with manual support compared to automated support. This finding was also not surprising. The activities involved in allocating time and resources and then developing the Gantt chart were more involved. Many of the planners could not determine a priori if they had gone over the time constraint until they graphed the time and task relationships using the Gantt chart. And as discussed previously, adjustments were easier to correct with automated support. Throughout the engineering design process, individuals and groups who planned their projects prior to beginning the project were required to hold status report meetings every thirty minutes. While these meetings were short, the purpose was for each participant to report their progress on the tasks for which they were responsible and the progress was recorded on the Gantt chart. Team design and reporting period were the significant effects for the average time elapsed in status report meetings. Groups had significantly longer status report meetings than individuals. The last status report meeting was significantly shorter than the first two meetings. Several of the individuals and groups had missing data points in third status meeting. The missing data was due to participants having completed the formal aspect of the design process (through detailed design). If participants were in the process of building or testing their designs, they were instructed to skip the meeting (however, the CPI and SPI were calculated for that point in time at the conclusion of the trial). Many participants allotted more time than they actually took for conceptual design and therefore, when they evaluated their progress in the first meeting, they were ahead of schedule. By the second meeting, they were in the midst of preliminary design, sometimes close to the end of preliminary design or at the beginning of detailed design. In the third meeting, many participants learned they had many tasks to complete to finishing the design process before time ran out. Therefore the final meeting tended to have a rushed sense without much discussion. 7.2 NASA TLX How was mental workload affected by team design and project support during the design projects lifecycle? Hypothesis 3: Mental workload was expected to be lower for groups than for individuals. Mental workload was expected to be greater in treatments with automated project support compared to treatments with manual support or without support. The analysis for the NASA TLX was first conducted as a comparison between groups and individuals. An average was calculated for each group which was then compared to the individuals ratings. 7.2.1 NASA TLX during Planning No significant differences were found in the analysis of the NASA TLX (TLX) during planning. The mean rating indicated a moderate workload for all treatments (groups: mean=12.3, sd=1.90; individuals: mean=13.4, sd=2.231). The TLX was similar regardless of the planning support (automated: mean=12.8, sd=2.139 and manual: mean=13.0, sd=2.18), also indicating a moderate workload. 168 Therefore, during planning the hypothesis that the workload for groups would be lower than individuals was not supported. Similarly the hypothesis for a difference between automated and manual project planning support was not supported. These results indicated that regardless of team design and method of supporting planning, the perceptions of mental workload were similar. Thus, automating the support tools did not add additional load nor did the use of the software save participants from additional loads, as may have been expected from the literature on function allocation (Hockey et al., 1989; Salvendy, 1981). 7.2.2 NASA TLX during Design During the design process, design phase and the interaction between design phase and project support were significant in the analysis of the NASA TLX. The TLX rating increased significantly over time as indicated by the differences between each design phase. There may be a cumulative effect contributing to workload perception over time. Though not tested, the average TLX for planning was lower than the TLX rating during conceptual design. This was an effect due to time pressure. The relationship will be explored later though correlations of TLX with time and perceptions of time. Previous research used the NASA TLX to evaluate mental workload during design (Harvey 1997). In Harveys study, differences within communication types were explored. However comparisons over time were not statistically tested. Mental workload averaged over the three different types of communication modes (face-to-face, video, and audio-only) in general was 6% higher in the first design period than in the third design period. For face-to-face groups, workload was 9% higher in the third design period than in the first design period. The results of the present study followed that of Harveys face-to-face groups; mental workload was 16% higher for participants in detailed design (the third design phase) than conceptual design (the first design phase). The studies diverged in that the face-to-face groups in Harveys study experienced a 6% decrease in mental workload between the first and second design periods; where as in the present study, participants experienced a 10% increase in mental workload. An important difference between these two studies was that Harveys participants worked on the design in three two-hour sessions. Approximately two days elapsed between each session. The results of Harveys study also provide support that the mental workload was accumulating over time. The finding based on design phase for the most part was valid when considering the interaction between project support and design phase. For all levels of project support, the TLX was significantly higher in detailed design compared to conceptual design. For manual and unsupported conditions, the difference was also significant between preliminary and detailed design. The difference between conceptual and preliminary design was only significant in treatments without project support. During conceptual and preliminary design, there was no difference between treatments with project support, but both had significantly higher TLX ratings than those without project support. For detailed design, there was no difference in the perceptions of workload based on the type of support. The differences based on project support during conceptual design were difficult to explain because there was no extra work imposed on participants with project support at this point in time. During conceptual design it was too early to hold status report meetings. One possibility was that there was a carry over effect from planning that did not exist for the participants who did not plan the project in advance. In preliminary design, the ratings were higher in treatments with automated support compared to those without support. During preliminary design the impact of the extra work due to the status report meetings may have been a factor. Participants without support did not hold status meetings. The meetings for the participants with automated support may have been what set the differences into motion. The participants with automated support had to quickly transition from a manually conducted design task into a computer environment in which progress data was entered into a computer. This transition between modes of operating requires attentional resources and might explain why the difference was significant 169 for those with automated support and not for those with manual support (Galliard 1993; Sanders et al., 1993). To summarize, based on the NASA TLX, the hypothesis that mental workload would be lower for groups than for individuals was not supported. The hypothesis that mental workload would be greater when participants used automated support compared to when participants use manual tools or those without project support was generally supported during conceptual and preliminary design, but not during detailed design. Components of the NASA TLX The NASA TLX was a weighted average of several rating scales: mental demand, physical demand, temporal demand, performance, effort, and frustration. To help understand from which aspects of the NASA TLX the participants felt demand, the variables were analyzed individually after a multivariate analysis of variance was conducted. Mental Demand In the analysis of mental demand, the interaction between design phase and team design was significant. Within conceptual design and preliminary design the mental demand was similar regardless of team design. However, within detailed design, the mental demand was rated significantly lower for groups than for individuals. This suggested that groups being able to divide the work amongst the members saved the members from additional mental loads. Groups rated mental demand differently from individuals. Initially groups started with large mental demands during conceptual design that significantly lessened during detailed design. This may be due in part to how groups tended to allocate the work. During conceptual design, without exception, all groups worked together on the problem statement, design requirements, and idea generation. During preliminary design, again many of the activities were conducted collaboratively. However, in detailed design, the work tended to be more independent. Note that these results mirrored the results from the job satisfaction question related to the perception of excessive work and will be discussed in the job satisfaction section. Physical Demand Design phase was significant in the analysis of physical demand. Physical demand was significantly lower during conceptual design compared to preliminary and detailed design. While a significant difference for physical demand might seem a odd in a task that was primarily an intellectual and creative exercise, these results were explainable based on the activities involved in each phase. During conceptual design, the main physical work associated was recording ideas on a piece of paper. During preliminary design, more movement was required as multiple design forms needed to be completed and the LEGOTM pieces used to prototype design concepts. Therefore, relatively speaking, while the rating was still fairly low, it was significantly higher in preliminary design than in conceptual design. While the nature of the work changed during detailed design into drawings and directions, there was still considerably more physical work involved compared to that required for conceptual design, thus the significant difference remained in detailed design compared with conceptual design. Even though there were significant differences the ratings indicated physical demand was low. Temporal Demand From the analysis of temporal demand, design phase was significant. The rating in each design phase was significantly different; over time the temporal demand became significantly higher. These results also mirrored the job satisfaction question regarding whether or not the participants had sufficient time to complete their task. As time elapsed the participants felt more pressure due to time. 170 In previous studies on workload, time pressure was often used as a fixed factor to determine the changes in perceptions of workload load (for example, Serfaty et al., 1993 and Beith, 1987). Understanding participants perceptions of time load was important in evaluating the affect of design on humans. Two measures involved in other aspects of this study were relevant here: one was the temporal rating scale and the other was the perception of sufficient time from the comfort facet in job satisfaction. In the analysis of perceived time, design phase was again a significant effect. All means from the multiple comparisons were significantly different. During conceptual design, participants agreed they had enough time to complete their task. By preliminary design, this perception had lessened to slightly agreed and by detailed design the perception was neutral. This result added support that over time participants perceptions changed regarding the design process; in this case the perception was the loss of available time. Performance There were no significant main effects or interactions in the analysis of performance. The lack of a difference indicated that groups did not feel additional pressure on their performance due to the presence of others. Working in groups has been sited as a potential concern related to increasing performance demands due to competitiveness or simply by knowing others were observing them (Hare, 1976; Gustafson et al., 1973). Effort Design phase and the interaction between design phase and project support were significant in the analysis of effort. Effort was significantly different in each design phase. As with temporal demand, over time the effort rating increased significantly. Therefore, the effort perception was in part due to a cumulative effect (again, while not tested, the mean rating in planning was lower than the mean rating in conceptual design). These results were similar to the TLX results. Another explanation of this result was due to the number of tasks required in each phase. Conceptual design required the least, preliminary and detailed design both required many tasks. During conceptual design, effort was significantly higher in treatments with automated support compared to treatments without project support. This effect may be a carry over from planning. Status reports did not begin until after conceptual design was complete. In addition, participants with automated support may have been burdened with the knowledge of the amount of effort required in future design phases (this was also supported by the lack of change over time for treatments with automated support). In preliminary design, effort was rated higher in treatments with automated support compared to those with manual support. This was caused by the automation, which previous studies found did not always reduce effort (Hockey et al., 1989). Another reason might be that those with automated support may be comparing their effort in design relative to what it might have been with automated design tools like CAD. During detailed design, the effort rating was similar for all project support conditions. By detailed design everyone was working hard to finish the task in the time allowed. Therefore it was logical that the ratings were similar and for the most part higher than in the previous phases. When holding project support constant and looking at the differences between phases, there were significant differences in treatments with automated support. The perceptions for those with automated support tended to be higher than that of the other conditions and remained stable over time. However, participants without project support rated effort significantly lower in conceptual design compared to preliminary and detailed design. Similarly, with manual support effort was rated higher in detailed design compared to conceptual and preliminary design. Again the argument is that to complete the design process on time, more effort was required in the latter stages due to the nature of the tasks and the time pressure that the participants perceived. 171 Frustration Design phase was significant in the analysis of frustration. Frustration was significantly lower during conceptual design compared to preliminary and detailed design. During conceptual design collaboration was encouraged and participants were not allowed to critique or eliminate any ideas. However as ideas had to be refined, compromises made, and details fined tuned the frustration rose. But it is important to point out that frustration only changed slightly between preliminary and detailed design. 7.2.3 Reflective NASA TLX The TLX rating was similar regardless of team design or project support and was consistently in the top quarter of the scale. Frustration was significantly different in the reflective evaluation. Groups experienced significantly less frustration than individuals when reflecting back over the entire process. This was initially surprising. When the design failed, one of the potential outcomes was for group members to blame each other; however, the testing process encouraged cooperation and quick problem solving, which appeared to be simpler in the groups than for the individuals. Therefore the groups could voice their disappointments to one another while individuals had no one with whom they could discuss their designs and the associated problems. The act of voicing concerns and problems and other issues can lessen frustration (Yaloms, 1985). 7.2.4 Planning Analysis of the NASA TLX for Groups by Role Again, as in the comparison between individuals and groups, there were no significant differences in the analysis of the TLX during planning. In the analysis of temporal demand, project support was significant. Those with automated support felt significantly less time pressure compared to those using manual support. This result was expected. The software combined several steps that those with manual support had to go through one at a time. In addition, changes could be easily made with the software. These results are also supported by the finding that those with automated project support finished planning significantly earlier than those with manual support. Project support was significant in the analysis of effort. Groups with manual support rated effort significantly higher than groups using automated support. This result made sense in that those with manual support had more cumbersome methods to iterate through. Often after the groups completed the Gantt chart they found their allocated exceeded the two hour limit. Therefore, they had to make adjustments using erasures and then adjusting the budget (which was often completed by the purchaser simultaneous to another member working on the Gantt chart). The software combined steps such that the Gantt chart was built automatically as the time estimations were entered and dependencies identified. When changes were made, the chart automatically updated all planning and tracking tools, for example the budget. Design Process In the analysis of the NASA TLX during design, design phase and the interaction between design phase and project support were significant. The TLX was significantly lower in conceptual design than in the other phases. This result was intuitive in that the output from conceptual design was relatively simple and the main activity was brainstorming. During preliminary and detailed design more outputs were required. During conceptual design, groups without support had a lower mean TLX rating than groups with project support. During preliminary and detailed design significant differences were not found. 172 With automated support, the TLX was significantly higher in preliminary and detailed design compared to conceptual design. A similar pattern occurred for groups without project support (the mean TLX score was lower during conceptual design compared to the other design phases). The finding that the rating was lower during conceptual design compared to detailed design was not surprising. What was surprising was the lack of a difference for the manual support over time. Additional work was required of groups with automated support compared to those without support. However the use of the computer to organize and store information compensated such that participants could be sensitive to changes in the requirements of the design process over time. In the previous analysis between groups and individuals this effect was masked by individuals because their workload was consistently heavy compared to the groups and impeded the individuals ability to be sensitive to changes in workload. Participants with manual support were not as sensitive to the changes in workload because the added activities required in planning and tracking the project desensitized them to the differences in design phases. In the analysis conducted on mental demand, physical demand, temporal demand, effort and frustration, design phase was significant. Mental demand was higher in conceptual and preliminary design than in detailed design. During conceptual and preliminary design, ideas were being developed, creativity employed, concepts were explored and tested, and the group members had to work together to come to agreement on several issues or, in some cases, they had to agree to disagree. During detailed design the groups could simply convert their ideas and prototypes from the previous phases into drawings and text. Therefore relatively little new thought was required during detailed design and the work could be divided amongst the members. As with the analysis between groups and individuals, physical demand was significantly lower during conceptual design compared to preliminary and detailed design. Although the difference was not significant, physical demand was slightly lower in preliminary design than in detailed design. Again, this result was anticipated. As discussed earlier, during conceptual design very little physical activity was involved; just recording ideas. However during preliminary design, the physical work increased to include the completion of several forms and prototyping designs and in detailed design, several drawings had to be completed in addition to textual descriptions and determining how various LEGOTM pieces fit together to clarify aspects of the drawing. Temporal demand was significantly lower in conceptual design compared to preliminary and detailed design. Based on the number of work products required in each phase and the points in time when the data were captured, this result was expected and supported the previous results discussed regarding time. During conceptual design, participants took their time in generating potential ideas and ensuring no one could come up with a new concept. However, in preliminary design, participants rushed more to fit everything in and leave enough time for detailed design. There was a slight increase between preliminary design and detailed design, though not significant. There was negative relationship between temporal demand and perceptions of time (which was relatively strong for preliminary design compared to the other phases). As temporal demand increased, the perception that there was sufficient time decreased. Similarly as temporal demand increased the number of time-related comments increased (which again was a relatively stronger relationship during preliminary design compared to the other phases). In the analysis of performance, the interaction between design phase and project support was significant. No differences were found between the design phases for participants with automated support. Participants with manual support had significantly higher performance rankings in conceptual design than in preliminary design. During conceptual design the performance was similar in all project support treatments. The mean indicated the performance rating was moderately high. In preliminary design, the performance 173 demand was higher for participants without support compared to those with manual support. During detailed design there were no differences. The general lack of significance during conceptual design indicated that participants had similar perceptions regarding the demand due to performance on the task. The difference for those with manual support in preliminary design might be due to a shift in the workload to the extreme that they did not care about what others thought of them. With automated support they were sufficiently aware of their progress and were more concerned about meeting their goals. From the analysis on effort, the rating was significantly lower during conceptual design compared to preliminary and detailed design. Based on the design products expected from each phase, this result was expected as discussed previously. Frustration was significantly lower in conceptual design compared with preliminary and detailed design. Frustration did not begin to set in until decision making tasks came to the forefront. During conceptual design, by nature of the task there was little room for conflict or need to negotiate because the goal was to come up with as many potential concepts as possible that would solve the problem. It was after the solution set had to be narrowed down and concessions made that the participants began to feel frustrations. In addition, the time pressure may be manifested as frustration (there was a slight positive correlation between frustration and temporal ratings, conceptual: r=0.146, p>0.05; preliminary: r=0.440, p<0.01; and detailed: r=0.343, p<0.05). Some of the group members became visibly agitated with one or more group members. In two cases, the person who experienced discontent with a member (or members) was the one who become disengaged, withdrawing into his or her own activities so he or she could completely ignore the group member causing the trouble. In considering the non-significant means there are several interpretations of the observations. For the TLX, mental demand, temporal demand, performance demand, and effort, the mean ratings based on project support level all fell on the upper half of the scale, indicating high demands where as physical demand and frustration were on the lower half of the scale, indicating low demands. This pattern was anticipated because design work was primarily mental in nature. While a difference in effort might have been expected for groups with status report meetings, due to the added layer of work, it did not create excessive perceptions compared to those who did not meet. All felt the time pressure equally. The frustration was relatively low, which contrasted with the early finding that individuals experienced more frustration than groups. But this reinforced the cathartic nature of voicing frustration amongst group members. Groups sharing an experience and voicing concerns relieved pressure from frustration. Reflective No significant differences were found in the analysis for the reflective NASA TLX. All of the mean ratings based on role or project support level were in the top third of the scale, indicating a relatively high workload. Upon reflection, there was a relatively moderate rating for mental workload and a relatively high rating for the temporal demand. The rating scale was not sensitive to the differences between demands in groups with project tracking requirements and groups without tracking requirements. While the rating for physical demand was low, there was a significant difference based on the type of project support. In automated conditions the rating was about half the rating in unsupported conditions. The difference between automated and manual support was not significant; however manual ratings were slightly higher than automated ratings. Participants with automated support had to rise and walk over to the computer to complete their status report meetings. However, because the Gantt charts were easier to read, they may have been better at limiting the time spent on activities associated with physical work (namely the prototyping). On the other hand, the manual groups often had difficulty reading their Gantt chart and did not pay as close attention to the amount of time spent on prototyping. In 174 addition, they had to sort through a variety of papers to find the appropriate reporting forms for the status report meetings. So the difference, though not significant, between automated and manual support is explainable. The expectation was because less work was required in of groups without support the physical load perceptions would not be as strong. Because the perceptions were relative and reflected back over the entire experience, participants with project support had a different context for comparison. The three contexts: 1) access to a computer to help with reporting and budget issues, 2) clear time expectations, and 3) clear task expectations in advance of the task, may have been on reflection an indication that relative to the other aspects of the TLX the physical demands was insignificant. Participants without support lacked the benefit of thinking through the design process in advance and did not have mechanisms in place to help them understand their progress relative to the design expectations in each phase. Therefore on reflection, this caused a different set of perceptions influenced by spending too much time in some activities and too little in others. 7.3 Job Satisfaction How was job satisfaction affected by team design and project support during the design projects lifecycle? Hypothesis 4: Job satisfaction was expected to be greater for groups than for individuals. Job satisfaction was expected to be similar in all project support treatments. Job satisfaction was measured using a faceted method that included comfort, challenge, and resources. This measure was based on Quinn and Sheppards (1974) faceted job satisfaction which was applied to a much larger population. In Appendix A.19 the reliability of the facets in this study were compared with the original study. Note that the reliability of the facets were a limitation in this study. 7.3.1 Job Satisfaction during Planning The perceptions of job satisfaction were similar regardless of team design and project support during planning. The responses used to determine job satisfaction were based on a seven point scale. When interpreting the mean satisfaction score and translating the scale by adjusting for the number of questions, the mean for groups indicated they were at the lower end of agreeing they were satisfied (mean=62, sd=4.78, and range for agree was 61-72). The score for individuals were more in the middle of the agree range (mean = 66, sd=8.69). The guidance from previous literature was mixed for this measure. When looking at satisfaction across varying group sizes, satisfaction was found to be unaffected by the treatment in several studies (Shaw, 1976; Hare, 1981; Hill, 1982; Meredith, 1997) but in others, satisfaction had been found to be better in small groups compared to large groups (Hackman et al., 1970). Taking this a step further, similar findings have been reported in studies comparing groups and individuals (Davis et al., 1987). However many studies used a global measure of satisfaction or one that related to specific aspects of the treatment (for example, the satisfaction with project support tools). In this study, since job satisfaction was measured based on three facets there was the possibility that differences between the facets averaged over the various treatment groups resulting in insignificant differences overall. When considering the mean differences for participants with support, there was no difference in job satisfaction (automated: mean = 66.28, sd=8.12; manual: mean=62.11, sd=5.48). While both scores were in the agree range for satisfaction, it was surprising that the differences were not significant. Again this might be attributed to the averaging effect of combining the scores from the three very different facets used to define job satisfaction. Because satisfaction had several facets that combined scores, by looking at each facet in closer detail revealed more about the participants perception of the work process. 175 Comfort No significant differences were found in the analysis conducted on comfort. The score indicated the participants agreed their work environment was comfortable regardless of team design and type of project planning support. Recall that comfort was determined based on responses to the following questions: perception of excess work, physical surroundings, sufficient time and ability to forget personal problems. The mean scores indicated that the workload was not excessive, the physical environment was relatively pleasant, and participants forgot their personal problems during planning. The only question with a significant difference related to job satisfaction and comfort was in the perception of time. Participants were asked if they had sufficient time to complete their task. Groups agreed they had sufficient time (mean=6.25, sd=0.754) compared to individuals who slightly agreed they had sufficient time (mean=5.25, sd=1.41). The literature on groups indicated that groups tend to require more time but that this was moderated by the type of task involved. Groups could work on parts of the planning process together (all worked on the scoping together) and then divide the work amongst group members, which might have influenced the perception that they had sufficient time, where as individuals had to accomplish all aspects of the planning task by themselves. Time savings for individuals occurred because they did not need to convince others to buy-in to their approach nor did they need to spend time discussing the various aspects of the project or who would be responsible for the various tasks. Challenge The mean scores for challenge indicated participants slightly agreed the job was challenging regardless of team design and project support. This was expected because the skills associated with planning tended to involve organizational skills and simple mathematics. Therefore the planning phase was simply formalizing processes they may have participated in previously without an imposed structure. The responses to questions used to determine challenge (developing ability, level of interest in the task, freedom in how to conduct the work, level of the problems difficulty, and visibility of the results) followed a similar pattern. The means indicated participants were neutral that they developed their ability during planning. There were slightly more favorable responses indicating tasks were interesting, they were given freedom to determine what they were supposed to do, the problem was sufficiently difficult and they could see their results. Resources The mean score for resources indicated participants agreed they had the appropriate resources to complete the task. Questions on access to the appropriate equipment, information, and a clear understanding of their responsibilities were used to determine if resources were sufficient. Participants agreed they had the appropriate information to carry out their task. As for understanding their responsibility and having access to the appropriate equipment, participants slightly agreed to agreed with the statements during planning. 7.3.2 Job Satisfaction during Design Design phase was a significant effect for job satisfaction during design. While job satisfaction was similar in conceptual and preliminary design, it decreased during detailed design. Even though the decrease in satisfaction was significant, the mean scores were all in the range of agree. Comfort Comfort was the only facet with a significant effect. Design phase was the significant effect in the analysis conducted on comfort. Comfort scores decreased significantly over time decreasing from agree to slightly agree. 176 Recall the questions used to determine comfort included perceptions regarding the excessive workload, physical surroundings, time perceptions, and personal problems. There were no significant differences in the analysis conducted on personal problems. All mean responses indicated participants agreed they could forget their personal problems during the design process regardless of design phase, team design, and project support. As time elapsed, the perception of sufficient time to complete the task decreased significantly. Initially after conceptual design, all participants agreed they had sufficient time to complete the task. After preliminary design the perception decreased to slightly agree. By the end of design, the perception was neutral. The perception of time was expected. The participants were reminded of time periodically so that they could finish the task on time. In the analysis of perceived excessive work, design phase was significant in addition to a three way interaction between all factors. Initially, after conceptual design, the participants agreed the workload was not excessive. This perception decreased significantly after each design phase until the perception was one of slight agreement. The three-way interaction needs to be considered before discussing implications of significant differences between each design phase. When comparing the effects within each design phase, there were no significant differences. For example, there were no differences for the treatment combinations between team design and project support during conceptual design. Within automated design, there were no significant differences for the treatment combinations between team design and design phase. There were significant differences for manual and unsupported treatments. For treatments with manual support, groups agreed the workload was not excessive in conceptual design; however in preliminary design this decreased to slightly agree. The means for preliminary and detailed design were similar. But because the variance grouping method impacted the degrees of freedom and the error terms, the difference between conceptual design and preliminary design was significant while the difference between conceptual design and detailed design was not significant. For individuals, the pattern was slightly different in treatments with manual support. During conceptual and preliminary design, the individuals agreed the workload was not excessive. In detailed design, the perception decreased to slightly agree. During preliminary design, groups slightly agreed the workload was not excessive while individuals agreed the workload was not excessive. This is one of the few times that indicated working in groups added to the participants perception of workload. To summarize, even with the significant differences, none of the mean responses indicated the workload was excessive. The lack of significance in the automated conditions was due to the software compensating and tracking activities that were done by hand in manual conditions. The unsupported conditions did not require the tracking activities. In the analysis of physical surroundings, the interaction between team design and project support was significant. For individuals, the physical surroundings were similar in all project support treatments. However, there was a slight decrease between automated and manual and between manual and unsupported. These differences can be explained by the level of organization and clutter on the work surfaces. Individuals with automated support had little clutter as they worked compared to those using manual support. Participants with project support were aware of the activities associated with their design process. Some had a method for organizing their work surface that those without project support lacked. The trend for groups is more difficult to explain. The groups each had to accommodate room for three people within a relatively small workspace. Those with automated support had their tracking tools stored on the computer and did not need to keep track of paperwork associated with tracking. Those with manual support needed to organize paperwork associated with planning and tracking in addition to design paperwork. Those without project support had less information and paperwork throughout the process which was reflected in the pleasant physical surroundings rating. While the score was slightly lower for those with automated support it was not significantly different. The lower score can be attributed to the computer located at the side of the design workspace. Participants moved between the design space 177 and the status report meeting space. The significantly lower score for participants with manual support compared to those without support was expected because the design papers and planning/tracking forms were piled in the same place. Challenge Challenge was similar regardless of design phase, team design or project support. Participants agreed the project was challenging. The questions used to calculate challenge included developing ability, interest in the problem, freedom within the work process, sufficient problem difficulty, and visibility of results. The insignificance of challenge was surprising considering that four of the five questions had significant effects. The interaction between team design and project support was significant in the analysis conducted on developing ability. Groups with support had significantly higher ratings compared to those with automated support. The opposite trend occurred for individuals, even though none of the differences for individuals were significantly different. In treatments without support, groups agreed they developed their abilities compared to individuals who were neutral. One suspicion was in the groups, the perception of developing ability was related to the role played. Sometimes participants assigned to the purchasing task verbalized unhappiness with that role. Conversely individuals, who had to accomplish everything from all three roles, felt the experience was richer because they were responsible for all aspects of the design. In the analysis on level of interest in the task, design phase was significant. In conceptual and preliminary design, the ratings indicated participants agreed the task was interesting. In detailed design, the rating decreased to slightly agree because of the nature of the activities. The aspects of the design process that involved forming new ideas included brainstorming potential concepts to satisfy the design requirements and taking two ideas and determining which might make the best design. In detailed design the idea was solidified and the activities were a matter of translating the design to paper via detailed drawings, manufacturing instructions, and a bill of materials. In preliminary design, participants were allowed to prototype, which enabled them to interact with the LEGOTM pieces and try out different concepts, which fully engaged the participants. In detailed design, not all of the participants were fully engaged. Often the purchaser waited for co-workers to finish their activities. In the analysis of how much freedom the participants had in approaching their work, design phase was significant. Initially, participants agreed they had freedom in approaching their work process. This perception decreased after preliminary design. In preliminary and detailed design, the perception of having freedom was lower compared to conceptual design. In conceptual design, the brainstorming session was a free-for-all. The only rule was ideas could be suggested but not eliminated. After conceptual design, participants had to ensure they produced the required outputs, which were more cumbersome. Traditional role playing tended to occur based on the participants assigned role in the latter phases. Although, participants were assigned to roles and given objectives for which they were accountable, they were not told they could only do tasks related to that role. Some chose to approach the task based on traditional roles, while others took a skills-based approach, helping each other as needed. Design phase was a significant effect from the analysis on sufficient problem difficulty. The rating was significantly lower in conceptual design compared to preliminary and detailed design. During conceptual design there was slight agreement the problem was sufficiently difficult and in preliminary and detailed design there was agreement the problem was sufficiently difficult. This can be due in part to the participants perception of time needed to complete each activity. There was evidence for this during preliminary design where positive correlations existed with the temporal rating, r=0.331, p<0.05, and with actual time spent in phase, r=0.331, p<0.05. During detailed design the correlations were negative and less than 0.1. Conceptual design mostly involved thinking while preliminary design involved more complex analysis skills and detailed design was committing the design to paper. 178 The analysis of participants ability to see the results of their work indicated design phase was significant. As time elapsed, the participants perceptions of their ability to see the results significantly decreased: in conceptual design participants tended to agree, in preliminary design they slightly agreed, and during detailed design they were neutral. The expected outputs from each design phase were concrete: conceptual design included a problem statement, list of design specifications/requirements, and a list of potential concepts; preliminary design required a trade off analysis and the selection of a design for embodiment; and detailed design required a series of detailed drawings, manufacturing instructions and bill of materials. After detailed design, questions remained because the participants had not tried to the build the system: 1) were there errors in the drawings and instructions? and 2) would the system work? These questions were now at the forefront because the only activities left after detailed design were to build and test the system. In the responses to questions used to determine challenge, conceptual design frequently had significantly higher scores compared with detailed design. The only contrary question regarded the level problem difficulty in which the opposite trend occurred. Resources There was little variation in the mean perception of resources. The participants agreed there were sufficient resources for the job regardless of design phase, project support, or team design. The questions used to determine the resource facet included access to the appropriate equipment, access to information, and clearly understanding responsibilities. In the analysis of access to the appropriate equipment, the three-way interaction was significant. Holding conceptual design constant and evaluating the combinations between team design and project support revealed a significant difference. Groups without project support rated their access to the appropriate equipment significantly lower than individuals. This was due in part to a lack of familiarity between the participants because this effect disappeared during preliminary and detailed design, where one might expect some dissatisfaction with the lack of design tools, for example, computer aided design software or a word processor. Trying to initiate or manage the discussion between a group of people who were unfamiliar with one another was a contributing issue. Holding project support constant and varying team design and design phase also revealed significant differences. Individuals with automated support rated their access to equipment higher in conceptual design compared to detailed design. In conceptual design, the primary activity was brainstorming to generate a list of ideas; pencil and paper were easily used to record ideas. Having used the computer during planning and using the computer to track progress might have created a slight dissatisfaction amongst the participants who conducted all aspects of the design process with pencil and paper. A similar pattern occurred for individuals without project support: the rating was higher in conceptual design compared with both preliminary and detailed design. The individuals without support easily recognized there were additional tools that might facilitate their design process when conducting the tradeoff analysis and embodiment activities. As discussed previously, groups during conceptual design indicated less satisfaction with the equipment compared with the later design phases. This might be a carry over of the group process which will be explored in Section 7.6. The mean responses ranged from slightly agree to agree that participants had access to appropriate information regardless of design phase, team design and project support. Similarly the mean responses indicated participants agreed they understood their responsibilities. In summary considering the significant differences for the questions used to determine comfort, challenge, and resources, challenge did not reflect any of the differences found in the individual questions. However, differences found for job satisfaction more closely reflected differences found for the individual questions. 179 7.3.3 Reflective Job Satisfaction Participants were asked to reflect back over their entire experience to evaluate their job satisfaction. The perceptions of job satisfaction were similar for all team designs and types of project support. In all conditions the mean score indicated that participants agreed they were satisfied with their job. Comfort: Similar to job satisfaction, in the ratings used to calculate comfort, the perception was in the agree range. There were no differences found in the responses regarding the participants perceptions of excessive workload, physical surroundings, personal problems and time perception. The means for excessive workload and having sufficient time to complete the task were slightly agreed, while the means for pleasant physical surroundings and ability to forget personal problems were agree. Challenge: The mean scores for challenge indicated the participants agreed the job was sufficiently challenging regardless of team design and project support. While looking at the response to questions used to determine challenge might not make sense from a statistical sense because the MANOVA did not indicate significant effects, there was one question with a significant effect. In the analysis of ability, the interaction between team design and project support was significant. The interaction was also significant in the evaluation during the design process. The trends in perceptions were similar except for two differences. In the reflective analysis, the difference between participants perception with automated support and without support was not significantly different. The other difference was the perception that individuals had regarding developing their ability was significantly higher for those with automated project support compared to those without project support. Participants indicated they agreed the problem was interesting and sufficiently difficulty, had freedom in approaching their work, and could see the results of their work. Resources: The mean score for resources indicated that the participants agreed they had the appropriate resources to complete the task. Similarly the participants agreed they had access to appropriate equipment and information, and clearly understood their responsibilities. The lack of differences in participants job satisfaction after reflecting back on the overall design process was expected. This set of measures was designed to capture a faceted measure of job satisfaction in long term jobs. The measure lost sensitivity over time which impacted the reflective evaluation in this study. 7.3.4 Analysis of Job Satisfaction by Role Data were reanalyzed with role as a factor to evaluate how the individuals role in the group affected their job satisfaction. Planning During planning, job satisfaction was similar regardless of role and project support as in the analysis between groups and individuals. The mean scores indicated participants were satisfied with the aspects of their jobs. Comfort: Project support was significant in the analysis conducted on time perception. Participants with automated support agreed they had sufficient time for planning (mean=5.889, sd=0.832) while those with manual support slightly agreed they had sufficient time (mean=4.611, sd=1.883). This difference was masked in the previous analysis that used a mean for the group value. As discussed previously in the temporal demand scale for the NASA TLX, the software packages used for planning combined several of the activities to develop a work breakdown structure and Gantt chart. In addition, changes could easily be made to the Gantt chart. Planning with manual support required more rework. With regards excessive work loads, physical surroundings, and ability to forget personal problems, the 180 responses were slightly more positive in treatments with automated support compared to treatments with manual support. No discernable patterns were observed for the responses based on role. Challenge: The interaction between project support and role was significant in the analysis of challenge during planning. Within levels of project support there were no differences amongst the roles. However, manufacturers with automated support had significantly higher challenge scores than manufacturers with manual support. Even though this result was significant there was no logical justification because during planning, the participants had not begun to conduct work based on their roles other than agreeing to task assignments. Analyzing planning by design roles may not provide any useful results. Assigning a distinct planning role to each member in a group was not possible in this study. After reviewing the video tapes, the groups approached planning very differently. For example, three of the teams with automated support worked so closely together on planning that their roles could not be distinguished (even to the point of taking turns entering information into the computer, or more commonly having one or two people control the keyboard while the third person controlled the mouse). In the manual groups the roles were more clearly defined, but often more than one person worked on a subtask in order to accomplish the task in the allowed time. Resources: In general, the mean scores for resources indicated participants agreed that resources were sufficient during planning. Design Process Design phase and project support were significant in the analysis of job satisfaction during design. Job satisfaction was higher in conceptual design than in detailed design. While job satisfaction was not significantly different between conceptual and preliminary design or between preliminary and detailed design, there was a slight decrease over time. Job satisfaction was significantly higher in treatments without support than in treatments with project support. The reason for this difference was the additional work due to planning and tracking contributed to negatively impact job satisfaction. Comfort In the analysis of comfort, design phase, project support and the three way interaction were significant. Comfort was higher during conceptual design compared to detailed design. This difference was expected based on the analysis for the effect of workload. There was evidence of a cumulative effect occurring as time pressure increased over time as will be shown in the analysis of the questions used to determine comfort. Comfort was significantly higher in the treatments without project support compared to treatments with project support. Again, this can be attributed to the extra work associated with status report meetings and time associated with planning. In conceptual design, in treatments with automated support, designers rated comfort significantly higher than purchasers. In conceptual design, purchasers without project support rated comfort significantly higher than those with automated support. These findings were odd in that role responsibilities typically did not enter into the picture during conceptual design. The purchaser was left with the responsibility of tracking progress in the treatments with project support; however conceptual design was typically over prior to the first status report meeting. The difference was attributed to the role expectations rather than to the overall job. During preliminary design, purchasers without project support and with automated support rated comfort higher than those with manual support. The reason for this difference was the additional work required for the status report meetings for those with manual support. The lack of a difference between purchasers with automated support and those without support may be due in part to the ease with which progress could be reported and updated using the software. On the other hand, purchasers with manual support had to first locate the Gantt chart, figure out how their group was doing and translate the progress to dollar amounts so that the CPI and SPI could be determined. These calculations were simplified with 181 the software tool used by groups with automated support. The groups without project support compared to groups with manual support had less work because they did not hold meetings. Two of the groups without project support spent more than 30 minutes on their conceptual design. Although during detailed design there were no differences between the treatment combinations of role and project support, the scores were slightly higher for participants without project support compared to those with project support. Again this trend was anticipated because the unsupported groups had less work to accomplish than the supported groups. Another way to consider the three-way interaction was to evaluate differences between types of project support. Designers with automated support had significantly lower comfort scores in detailed design compared to those in conceptual design. Again the reason for this effect was the cumulative affect of the workload and time pressure. Designers without project support showed a similar pattern to those with automated support. The main difference was the scores were higher in both conceptual and preliminary design than in detailed design. The sudden decrease in the designers rating was due to the lack of planning. Designers were responsible for three detailed drawings, which were the most commonly shared tasks when groups were running out of time. Groups that did not plan their strategy inadvertently placed more pressure on their designers by not leaving them enough time. Manufacturers with manual support rated comfort significantly higher during conceptual and preliminary design compared to detailed design. Again this resulted due to time pressure. For purchasers using manual support, comfort was significantly higher during conceptual design than in preliminary design. This decrease was associated with the first status report meeting. The opposite trend occurred for purchasers without support; their scores were significantly lower in conceptual design compared to the other phases. One explanation was purchasers took longer to become engaged in the project. Over time as the purchaser became more engaged, they forgot their surroundings and personal problems. The last way to consider the three way interaction is to hold role constant and evaluate differences between design phase and project support. Designers in all treatments had the trend that comfort was highest in conceptual design and lowest in detailed design. The only significant differences were between conceptual and detailed design in treatments with automated support and without support. The manufacturers followed a similar pattern to the designers though less pronounced. Manufactures with manual support had significantly higher comfort scores in conceptual and preliminary design than in detailed design. Design phase, project support and the three way interaction were again significant for the analysis of excessive work. In conceptual design, participants agreed the work was not excessive, but by the end of detailed design they only slightly agreed. This is evidence supporting the conclusion that over time there was a cumulative effect impacting the perception of the task. Participants without project support agreed the work was not excessive; however those with manual support only slightly agreed, which was a significant difference. The mean response of participants with automated support agreed the work was not excessive. Groups without project support had less paperwork and required tasks during their design process. The other extreme was manually supported groups reported, calculated and recorded all of their tracking reports using pencil and paper. This added work made the difference. The response for groups with automated support was in between the two extremes and was not significantly different from either one. In treatments with automated support, the additional responsibilities accounted for the slight decrease in the mean compared to those without support. Because groups had software to manage tracking, the perceptions were not different from the groups that did not have to track progress. Similarly, the responses from those with manual support indicated a slight increase in the workload. This is evidence that automating the tracking process worked. Some of the details the software handled included automatically calculating the percentage of design phases completed based on specific activity completion, graphically representing the tracking, and calculating intermediate costs and the ability to quickly visually assess that the project was on schedule. 182 Purchasers with automated support during conceptual design slightly agreed the work was not excessive compared to designers and manufacturers who agreed work was not excessive. This pattern was also observed in the comfort analysis, which was not expected because the activities associated with conceptual design were not labor intensive. However, purchasers were not completely engaged in the activity in the initial phase as already pointed out. The mean score for purchasers with automated support was also significantly lower than the mean score for those in the other treatments. Another issue adding to this effect was the transfer of attention from the computer during planning to pencil and paper during design. This difference was not reflected by designers and manufacturers. During preliminary design in treatments with manual support, purchasers were neutral that the workload was not excessive, which was a significantly lower than that for designers and manufacturers, who agreed with the statement. Purchasers perceived the workload was more excessive because they assumed responsibility for the status report meetings. Preliminary design involved one or two status report meetings. Purchasers with manual support discerned additional loads and it influenced the excessive workload perception and comfort level. However, the NASA TLX scales did not show differences. This observation was further reinforced taking into account purchasers without project support and with automated support agreed that work was not excessive. The support added by the computer compensated for the difficulties associated with manually tracking. The absence of status report meetings in the treatments without support eliminated a layer of complexity. Excessive workload was significantly different based role during detailed design. Manufacturers with manual support were neutral about excessive workload. The manufacturers perceptions were significantly different from the designers who agreed workload was not excessive. The manufacturers mean score was significantly lower than the purchasers who agreed the workload was not excessive in automated conditions. The evaluation of individual perceptions of workload was a relative issue. Exploring this idea, manufacturers with automated support had a decrease in the mean score over time indicating a slight increase in the perceived workload. Manufacturers with manual support had similar responses during conceptual and preliminary design with a sharp decrease during detailed design. These results corresponded to the nature of the activities. During preliminary design, the manufacturers main responsibility was to prototype with the LEGOTM pieces. Prototyping was different compared to detailed design work, which involved writing detailed instructions for use with drawings to build the system. The problem with this explanation arises from the manufacturers without project support who had a trend of slightly decreasing scores, which was not significant from phase to phase. The added activities associated with status reports contributed to this difference. Manufacturers with project support did not take the lead in tracking, but the interruption caused by meeting was a nuisance. These meeting occasionally were accompanied with comments like not again or do I have to participate. Another explanation could be from the responsibility assignments during planning to promote accountability during design. The groups without support seemed more flexible helping each other compared to the other groups that remained more rigid in their role definitions. Later in this chapter, the exploration of group process involving adaptability and cooperation behaviors provide additional insight. Designers without project support over time perceived significant increases in excessive work. As discussed earlier in the section on comfort, this difference was attributed to a lack of planning and leaving insufficient time for the designers to complete drawings or lack of mental preparation because differences were not significant in treatments with project support. There were no significant differences in perceptions of excessive work over time for purchasers without project support. This result seems logical because purchasers without support did not have added responsibilities for meetings. The purchasers with manual support perceived more excessive workloads during preliminary design while purchasers with automated support perceived more excessive workloads during conceptual design which dissipated by the end of preliminary design. 183 Project support was significant in the analysis conducted on physical surroundings. The physical surroundings were rated significantly lower for treatments with project support compared to those without support. In treatments with automated support, the lower rating was caused by the need to move between the work table and the computer table to conduct status report meetings. In addition three participants had to share one computer. Although the mean scores were significantly different, the interpretation indicated they agreed the physical surroundings were pleasant in all conditions. Participants with manual conditions had double the paperwork of those without support. Therefore groups with manual support were often fumbling for the appropriate paperwork. Design phase was significant for the analysis of sufficient time to complete the task. The participants agreed there was sufficient time to complete the task during conceptual design. Scores significantly decreased in preliminary design to slightly agree and neutral in detailed design although not significantly different. Because of the subjectivity of the perceptions, these results could have been expected. The groups had two hours from the beginning of the design process to complete their project. Therefore, taking 15-30 minutes to complete conceptual design left plenty of time to complete the design project. However, many groups left little time for their detailed design activities and had to rush to finish. Phrases like hurry up or we are running out of time were common amongst group members. There were no significant differences in the analysis of personal problems. The mean response indicated participants agreed they could forget personal problems while working on the design task. Challenge The participants agreed the job was challenging and the mean scores between factors were not different. Perceptions on developing personal ability were significantly different depending on the type of project support. Participants with automated support slightly agreed they developed their ability compared to participants without project support who agreed they developed their ability. The analysis on the level of interest in the task revealed a significant difference over time. The participants agreed the task was interesting in all phases, but the mean was significantly lower in detailed design compared to conceptual and preliminary design. This result is explained by the creativity and collective decision making activities during conceptual and preliminary design. The detailed design was routine in nature, converting ideas to paper to be refined for others to build the system. The interaction between role and project support was significant for the level of interest in the task. In treatments with automated support, designers had significantly higher mean scores compared to purchasers. The opposite occurred in treatments with manual support; purchasers had higher scores than designers and manufacturers. There were no differences in perceptions between roles in treatments without project supported. The difference between roles with automated support was because of the rigorous approach to role responsibilities. The purchasers job, if strictly adhered to, consisted of tasks related to cost estimates and bills of materials; contrasted with the designer who was charged with converting the design concept into a reproducible system through sketches and detailed drawings. In treatments with manual support, purchasers rated their interest higher than the other roles. The status report activities were mainly handled by purchasers because their responsibilities were not as time consuming as responsibilities of other members. The additional work involved with holding the status reports for purchasers with manual support enriched their job. For treatments without project support, the difference reflected a relaxation of the role structure also enriching the purchasers job with opportunities to help with drawings and instructions. Design phase was significant in the analysis on the ability to see the results of the work. As time elapsed, the participants were able to see the results of their efforts emerge. All participants slightly agreed/agreed the problem was sufficiently difficult and they had freedom to design their work process. 184 The overall effects of the individual questions revealed the problem was sufficiently challenging to engage their interest. The major shortcoming may have been in the assignment of roles. The role of purchaser was not interesting to most engineering students. Future consideration for controlled research studies should consider using a business major for this role or to redesign the role to be more comprehensive and interesting. Resources In the analysis on resources, the interaction between project support and role was significant. Designers had higher scores compared to purchasers in treatments with automated support. In manual treatments, purchasers had significantly higher scores compared to manufacturers. Purchasers without support strongly agreed they had adequate resources, which was not significantly different from purchasers with manual support but was significantly higher than the scores for those with automated support. The design tools which were manual influenced this result. Manufacturers had higher resources scores than the purchasers during conceptual design in treatments with automated support. During conceptual design, purchasers with automated support had lower scores compared to those without support. The purchasers through out the analysis of job satisfaction had lower scores during conceptual design that increased over time. In detailed design with automated support, designers had a higher mean resources score than purchasers. Purchasers with manual support had a higher mean score than designers and manufacturers with manual support. During detailed design, manufacturers without support had higher mean scores compared to those with manual support and purchasers without support had higher mean scores than those with automated support. For designers with manual support, resources were rated higher in conceptual design than in preliminary design. For manufacturers with automated support, resources were rated lower in detailed design than in conceptual design. A similar pattern occurred for manufactures with manual support in which resources were lower during detailed design than in the earlier design phases. A repeated pattern occurred for purchasers. In all design phases, the resource score for treatments with automated support were significantly lower than those for treatments without support. Purchasers with automated support understood weaknesses in the project software. They could use the computer as a tool for planning and tracking but did not have access for designing. Purchasers could only use the software to calculate intermediate values for the performance indices after which they needed to complete the calculation by hand. Recall purchasers often took the lead in this activity because they had the least work. The individual questions were evaluated to explain the differences in the analysis of resources. There were no significant differences in analysis of equipment, information and responsibility. Two questions relating the competence and helpfulness of the group members provided insight because significant effects and interactions existed. The means for helpfulness were significantly higher in treatments without support than in the other treatments. With automated support, designers and manufacturers agreed their group members were competent; however purchasers slightly agreed, which was significantly different from designers and manufacturers. In treatments with automated support, designers slightly agreed their group members were competent compared to purchasers who agreed and manufacturers who were in the middle. Considering each role across the types of project support, there were no differences for designers or manufacturers. Purchasers in unsupported conditions strongly agreed their group members were competent compared to purchasers in automated conditions who slightly agreed. A reason for the difference was attributed to inherent competitiveness. The purchasers had a supporting function in the design process and many purchasers seemed to emerge as the group coordinator. In two unusual cases, 185 the purchaser became very negative towards group members. In one case the purchaser was frustrated by the definition of the role and did not take initiative to break out of the role to help group members when they needed help. In the other case, the purchaser was angry with an unengaged group member and literally took over the designers role in addition to maintaining the purchasers role. As the design process progressed, the participants scores decreased significantly in the evaluation group member helpfulness. Initially participants strongly agreed members were helpful. The mean score was significantly lower by the end of detailed design. During conceptual design the goal was to determine as many potential ideas as possible. The groups worked together to discuss and explain ideas so all members understood the ideas. During preliminary design, some aspects of the task were worked on together, for example, narrowing the ideas down to two and discussing the strengths and weaknesses of the two concepts, and finally selecting a single idea. These were the decision making aspects of the task. However, many of the activities were conducted independently, for example, writing an explanation of the concept, sketching the concept and estimating the costs. During detailed design, work was primarily independent. Even if two individuals shared a task, they tended to work on their part independently with the exception of asking clarifying questions or ensuring they correctly captured various aspects of the design. Designers rated their group members more helpful than purchasers in treatments with automated support. Purchasers rated their group members more helpful than designers or manufacturers in manual conditions. Designers rated their group members more helpful in treatments with automated support and without support compared to treatments with manual support. Purchasers rated their members more helpful in treatments without support and with manual support compared to those with automated support. The pattern of these results closely followed the results of competency. Reflective The analysis on job satisfaction assessed to reflect over the entire experience was not significant. Comfort: The mean scores for comfort were similar across all treatments. The responses to the questions (workload, physical surroundings, perceived time and personal problems) were similar. Challenge: The participants agreed the various aspects of the task represented sufficient challenge. The MANOVA did not indicate significant differences for the reflective questions used to calculate challenge. However, for one question the variance in the response violated homogeneity of variance which was cause to question the validity of the MANOVA. Several significant differences were found in the questions used to determine challenge. Caution is needed in interpreting these results. Project support was significant in the analysis conducted for developing ability. Participants using automated support slightly agreed they were able to develop their ability compared to participants in without support who agreed. From the analysis on reflective interest, in treatments with automated support, purchasers slightly agreed the task was interesting while manufacturers and designers agreed the task was interesting. Purchasers with manual or no support agreed the task was interesting while those with automated support slightly agreed. This was attributed to the design of the job based on the role. In treatments with automated support, because of the formalized accountability and role assignments, participants conducted tasks strictly within the limits of their roles unless other members invited help. The interaction between project support and role was significant in the analysis of the question if the problem was sufficiently difficult. Purchasers with automated support caused the significant difference. They slightly agreed the problem was sufficiently difficult compared to those with manual or no support who agreed there was sufficient difficulty. Purchasers had lower scores than manufacturers. Participants agreed they had freedom to design their approach to work and saw the results of their work. 186 Resources: The participants agreed they had the appropriate resources for the job. There were no significant differences in responses to questions used to calculate the resources score. In summary, job satisfaction was insensitive to the independent factors during planning and upon reflection. The perceptions ranged from slightly agree to agree based on the facets used to determine job satisfaction. Therefore in terms of job design, evaluating overall job satisfaction was too broad to determine how to best organize individual aspects of the job. Individual questions used to calculate job satisfaction provided important feedback for use in determining guidelines for the work process. 7.4 Relationship between Job Satisfaction and Workload or Performance Job satisfaction was linked to workload in earlier research. Too little work and/or challenge caused boredom as a result of unused mental capacity. Too much challenge or work overwhelmed mental capacity. In either case, the result was dissatisfaction in prior research (Locke, 1976). Previous research indicated a negative correlation existed between lack of job satisfaction, workload, and frustration (Spector et al., 1988; Jex et al., 1992). These aspects were captured in this study and the relationships were explored to test the previous findings. During conceptual design, there was a significant positive correlation between mental demand and job satisfaction (r=0.376) and a negative correlation between job satisfaction and frustration (r=-0.336). During preliminary and detailed design there was a negative correlation with frustration (preliminary: r=-0.424; detailed: r=-0.513). Reflectively there was also a negative relationship between job satisfaction and frustration (r=-0.433). The correlation during planning was negative and not significant (r=-0.320). These results confirm a relationship between frustration and job satisfaction as previously reported. Previous studies also found positive correlations between job satisfaction and performance. In an analysis of twenty studies between the 1970s and 1980s, the correlation between job satisfaction and individual performance was 0.31 for professionals and 0.15 for nonprofessionals. From a meta-analysis conducted by Iaffaldano and Muchinsky (1985), the correlation between job satisfaction and performance was 0.146 indicating a small positive correlation between job satisfaction and individual performance. The data in the Iaffaldano analysis came from seventy articles published between 1944 and 1983. In this study there was a small positive correlation between cost effectiveness (performance) and job satisfaction. There were also small positive correlations between job satisfaction and system effectiveness, reliability, and design costs. Small negative correlations were found between job satisfaction and life-cycle cost, material costs, and the number of errors. With the exception of the design cost, these relationships indicated job satisfaction increased relative to performance. An example of a negative correlation showing the relationship is a decrease in life-cycle cost corresponded to an increase in performance. These results confirmed results from previous studies and in many cases showed stronger relationships than previously reported. The earlier work occurred in real work settings compared to the controlled environment in this study. 7.5 Group Workload Planning The mean ratings for the value of group interaction, difficulty of interacting, degree of cooperation and overall group workload were similar. The value of group interaction and degree of cooperation were each rated toward the high end of the scale and the difficulty of group interaction was rated toward the low end of the scale. The overall group workload was moderate. 187 Design Process Design phase was significant in the analysis of value of group interaction. The group interaction was not as valued during detailed design compared to the other phases. During conceptual design, participants had to work together to develop list of ideas. More independent work occurred during preliminary design except decision making and problem solving were group activities. An example of a group decision was reducing the list of ideas to two. An example of problem solving occurred while prototyping concepts. The entire group made suggestions on which LEGOTM pieces to use and how to put the pieces together. These types of activities lent themselves to group interaction. On the other hand, in detailed design, the work was more independent because they could divide labor amongst the group members. The group members depended on each other to complete their tasks so the group would have a completed product. They did not work directly together as much as in the first two design phases. Design phase was a significant effect in the analysis of the difficulty of group interaction. The ratings indicated the difficulty of interacting was relatively low. There was more difficulty interacting during preliminary design and detailed design compared to conceptual design. Anecdotally, at least two groups had members who asked many questions during detailed design and were rebuffed for frequently interrupting. The activity associated with conceptual design was not expected to have much difficulty because participants were asked to generate ideas and leave the evaluation until preliminary design. The difficulty arose in preliminary design from participants to convince other members that their design was better. In addition, there were several interrelated activities required in preliminary design and detailed design. This added to the complexity in the group work process. The relationship between the difficulty of group interaction and group behaviors are discussed in the last section of this chapter. There was a negative correlation between difficulty of group interaction and perceptions of group member helpfulness and competence. The correlations suggested as group members perception of competence and helpfulness decreased the group interaction became more difficult. The analysis on the degree of cooperation revealed a significant difference for design phase. The degree of cooperation was significantly higher during conceptual design compared to the other design phases. In conceptual design, all members worked together on listing the design requirements and generating design ideas. However, in preliminary design the work was a mixture of independent and collaborative work and in detailed design the majority of the phase consisted of independent work with some verifications or requests for assistance. Design phase was also significant in the analysis of overall group workload. The explanation for this result was similar to the explanation for the other group rating scales. The group workload during conceptual design was lower than the workload in preliminary design and detailed design. The ratings for the latter two phases indicated the workload was perceived to be higher but were not extreme ratings. Comparing the results of this study to the initial study that used the rating scales found a few similarities. In the initial study by Beith (1987), teams were subjected to either high stress or low stress scenarios. In teams that were allowed to interact, the difficulty was 141% higher in high stress scenarios than low stress scenarios. The value of the interaction was 17% higher in the low stress scenario for interacting groups. And the overall workload was 99% higher in the high stress scenario than the low stress scenario. The stress level was manipulated as time stress. In the present study, stress level was not directly manipulated. However, as each design phase was completed the temporal demand rated by participants increased. By the end of the design process, temporal demand increased by 45% from the demand in conceptual design. Therefore, the argument can be made that there was pressure due to time. The value of the demand was 25% higher in conceptual design (analogous to the low stress scenario) than in detailed design. And the difficulty of the interaction was 39% higher in detailed design (analogous to the high stress scenario) than in conceptual design. The overall group workload was 38% higher in detailed design than in conceptual design. The trends between the two studies are similar in that the 188 difficulty and the overall workload were greater in the cases where there was higher time pressure. And the value was greater in the cases where there was lower time pressure. Reflective Project support was significant in the analysis conducted on overall group workload reflecting over the entire experience. The overall group workload was significantly lower in treatments with automated support than in treatments without support. As will be explored in the group process section (Section 7.6), groups with automated support did not overtly interact as much to attend to group process resulting in lower group workloads than the others. This finding suggests the computer provided the coordinating function that humans provided in unsupported and manual conditions. This was also supported by the slightly higher rating for participants with manual support compared participants with automated support. That the difference was not significant from treatments without support indicated the manual support played a role. Including planning and tracking slightly lowered the workload in groups with manual support and significantly lowered workload in groups with automated support. Since manual conditions were not significantly different from automated, the notion that project tools provided support to the group process was reinforced. In a previous study that used the group workload scales a strong positive correlation was found between the NASA TLX and the overall group workload (Beith, 1987). The relationships were positive but only planning was significant. 7.5.1 Planning Group Workload Evaluated by Outsider Observers During planning, the outside observers evaluation of group workload scales revealed no differences between the factors. While the differences were not significant, the value and the degree of cooperation were rated slightly higher for groups with automated support compared to groups with the other kinds of support. The overall group workload was slightly higher for groups with manual support compared to those with automated support. The ratings for value, cooperation, difficulty and overall workload were moderate. Design Process The outside observers rated the difficulty of group interaction significantly different depending on the design phase. The group interaction was more difficult during preliminary and detailed design than in conceptual design. This was a reflection of the number of activities and the nature of the activities associated with each design phase. During conceptual design, most participants were supportive while brainstorming ideas together. During preliminary design the members had to accomplish more tasks, some of which were interrelated. For example, the cost estimate could not be completed until the designer had a rough idea of what the concept would look like and the manufacturer had started working out which materials were necessary. Detailed design was significantly more difficult than conceptual design, but only slightly less difficult than preliminary design. Detailed design involved many activities. Because of the decisions made during preliminary design, some of the detailed design activities were easily conducted independently. Communication during detailed design was imperative. If the designer or manufacturer decided to make a change that deviated from their original decision, they needed to communicate those changes. Otherwise the bill of materials, manufacturing instructions, and detailed drawings would not match. Anecdotally, in the building phase, if there was an error, it was accompanied by the comment from the designer along the lines of, I changed that, with the designer being the only one aware of the change. While many aspects of detailed design were independent, information sharing contributed to the higher ratings. 189 Design phase was a significant effect in the analysis of the overall group workload as evaluated by outside observers. Overall group workload was rated higher during preliminary and detailed design compared with conceptual design. The reasons outlined in the discussion for difficulty of the interaction were also true for the overall group workload. In addition, unlike the difficulty of the group interaction, the extreme difference was found to occur between conceptual design and detailed design. So while there remained a similarity between preliminary and detailed design, the nature of the trend was an increase. This suggests there was a cumulative effect associated with the external observations of group workload. Reflective The ratings of the group workload scales evaluated by the outside observers taking into consideration all aspects of the experience were similar to the perceptions captured during planning and design. However, there were no significant differences. The lack of difference was expected because of the nature of this study. However, the trend in means were interesting because there was a suggestion that planning and tracking with the automated support could reduce the overall group workload even if just slightly. 7.6 Critical Team Behaviors Planning Project support was significant in the analysis of all ineffective team behaviors during planning. Treatments with manual support had more ineffective behaviors observed than those with automated support. To put this into context, less than one ineffective behavior per group with automated support was observed compared to one per group on average with manual support. In the analysis of all effective team behaviors, project support was again significant. Treatments with manual support had more effective behaviors on average than treatments with automated support. There were almost twice as many effective behaviors observed in manual treatments than observed in automated treatments. The results of the analysis of overall positive and negative behaviors suggested that the treatments with automated support discouraged interaction of any kind (effective or ineffective) amongst the participants. Because all participants could see the computer screen while the data was entered, unless they disagreed with something, there was little need to verbalize. The results suggested the project planning and tracking software was a tool that assisted with managing group process during planning. This finding is important because planning consisted of organizing and coordinating activities. For the manual conditions on the other hand, because people worked on different aspects of the planning process, they had to overtly share their work which required more group interaction. Often these individuals did not share the same planning space or tools. For example, one person worked on the Gantt chart while another completed the budget. Groups with automated support shared the same planning space through out the entire planning process. The specific behaviors will be discussed in more detail to understand which were more important. Effective coordination and giving feedback behaviors both had significant differences based on the type of project support. Treatments with automated support had an average of two effective coordination observations per group, which was significantly lower than the average of three for those with manual support. Treatments with automated support had fewer effective giving feedback behaviors observed (an average of one) compared to those with manual support (an average of two). The computer software provided a coordinating mechanism so not as much interaction was required in groups with automated support. On the other hand, as suggested in previous research, the computer may have impeded group process by consuming additional attentional processes (e.g., Hockey et al., 1989). 190 The remaining behaviors did not have many observations and several in fact could not be analyzed using traditional methods. In all of observations reported for the various non-significant factors, the observations translated into less than one observation per group on average. These results suggested that blending the two support approaches for planning might be useful. To generate more discussion and attend to group process initially manual methods would be effective, followed by a change over to the computer for entering information into the Gantt chart which would result in a better overall plan. With the proposed type of structure, the initial part of planning would generate more discussion and enable the participants to interact and using the software would help to schedule the activities, make changes, and track progress. This would be a logical advance in developing guidelines for project management. Design Process Design phase and role were significant in the analysis of all ineffective behaviors. More ineffective behaviors were observed during preliminary and detailed design compared with conceptual design. On average there was less than one ineffective observation per group during conceptual design compared to at least one ineffective behavior observed during both preliminary and detailed design. In addition, the purchaser averaged at least one ineffective behavior per group during the design process compared with the designer and manufacturer who each averaged less than one. Role and design phase were again significant effects in the analysis of all effective behaviors. More effective behaviors occurred during preliminary design and detailed design compared to conceptual design. In conceptual design, groups averaged four effective behaviors compared to eight for preliminary design and six for detailed design. Purchasers averaged seven effective behaviors per group during the course of the design process compared to five for the designer and manufacturer, which was a significant difference. This result suggested that purchasers assumed the responsibility for managing the group process (or at least attending to the process). The results during the design process were a reflection of the time spent in each phase (conceptual design was about half as long as the other design phases) and also the nature of the tasks involved in each phase. Because the task during conceptual design was as a group to determine the design requirements and concepts, the conversation focused on sharing ideas. The process behaviors centered on initiating the design process followed by determining if the group had exhausted their ideas and could move into the next design phase. During preliminary design and detailed design, sometimes statements were used to coordinate activity or assistance was offered if someone looked overwhelmed. The nature of the roles also influenced these results. Once the ideas were solidified, the designer could work independently with little input from the others except for checking with the manufacturer from time to time to ensure their concepts converged. The purchaser depended on the information from both the manufacturer and designer. The purchaser had fewer complex responsibilities than the designer and manufacturer and could easily attend to the managing the group process. In the analysis of ineffective communication behaviors, the interaction between design phase and project support was significant. In conceptual design, there was more ineffective communication observed in treatments without project support compared to treatments with project support (while the difference was significant, the ineffective communication observations were still less than one per group). This might be due to the familiarity that grew amongst the group members during planning. For preliminary design, treatments without project support had less ineffective communication compared to treatments with automated support. In treatments with automated support, more ineffective behaviors were observed in preliminary design than in conceptual and detailed design. In treatments without support, more ineffective communication behaviors were observed during conceptual design compared to preliminary and detailed design. There were few effective communication behaviors observed (a total of three in conceptual design, seven in preliminary design, and six in detailed design; six for automated, five for manual, and 191 five for unsupported). There were no ineffective cooperation behaviors observed during conceptual design. There were a total of seven observations in preliminary design and ten in detailed design. Four were observed in treatments with automated support, five with manual support, and eight without support. In the analysis of the effective cooperation, role and design phase were the significant main effects. More effective cooperation occurred in conceptual and preliminary design than in detailed design. On average, the purchaser had one more effective cooperation behavior than the others. Design phase was significant in the analysis of effective coordination observations. On average there was one less effective cooperation behavior during conceptual design than in preliminary or detailed design. There were no ineffective giving feedback behaviors observed in conceptual design and only two in preliminary and six in detailed design. Two of the ineffective observations occurred in treatments without project support, three with automated support and three with manual support. From the analysis of the effective giving feedback, design phase was significant. Less than one effective behavior was observed on average (converts to about one observation in every two projects) during conceptual design compared to at least one per project during preliminary and detailed design. Only one instance of ineffective acceptance of feedback was observed in conceptual design and two were observed in preliminary design, while none were observed in detailed design. One each was observed in unsupported, manual and automated conditions. From the positive acceptance of feedback there were no observations in conceptual design and a fraction in the other phases. Manual support and detailed design did not have any observations of ineffective adaptability. Only a fraction was observed in the other treatment combinations. On the other hand, design phase, project support and role were significant effects in the analysis of effective adaptability behaviors. There were no effective behaviors observed during conceptual design compared to approximately one observation for every two projects in preliminary and detailed design. The purchaser exhibited more effective behaviors than the other two positions (although this translated to about one observed behavior in every two projects). There were fewer effective adaptability behaviors observed in treatments with automated support compared to the other treatments. The behaviors focused around the purchaser. When significant differences were observed the purchase was the role with more observations. This reinforced the conclusion that the purchase attended to group process more than designer or manufacture. Fewer behaviors were observed in treatments with automation because verbalization was not necessary to coordinate the project. 7.6.1 Correlations with Critical Team Behaviors Correlations with Critical Team Behaviors during Planning Because the participants came from a diversity of backgrounds their demographics were correlated with the critical team behaviors. The only interesting correlation was the number of self reported design projects increased with the increased number of effective communication behaviors (r=0.746, p=0.005). This suggested experience improved communication skills. For the Gantt chart score, as the number of ineffective observations related to communication, cooperation, giving feedback, and team spirit and morale increased, the score of the Gantt chart decreased. Therefore the relationship between all ineffective behaviors and the score was predictable. This finding has an implication that poor group process detracts from planning. There was a negative relationship between effective observations of giving feedback and overall effective behaviors scores. This reinforced the finding in that too much interaction, regardless of effective or ineffective, distracts from planning. 192 There was a negative relationship between several critical team behaviors and mental demand. In other words, as mental demand increased the number of effective behaviors related to adaptability decreased and the number of ineffective behaviors related to communication, cooperation, coordination, giving feedback and team spirit and morale decreased. Similarly there was a negative relationship between mental demand and the overall number of ineffective behaviors. This finding suggests as mental demand increased participants did not have as much energy or mental capacity to contribute ineffective team behaviors. A correlation existed between temporal demand and effective giving feedback and team spirit and morale. This suggested that the groups under more time pressure reacted by encouraging their group members. Frustration increased with ineffective behaviors including communication, cooperation, coordination, team spirit and morale, and overall ineffective behaviors. This suggested the groups could perceive the ineffective group behaviors during planning and the manifestation was frustration. Finally the overall TLX ratings were related to giving effective feedback. As the amount of effective giving feedback increased, so did the TLX rating. As comfort decreased, the number of effective acceptance of feedback behaviors slightly increased. Similarly as comfort decreased, there was an increase in ineffective communication, coordination, and team spirit and morale. The relationship between effective team spirit and morale behaviors and comfort was positive indicating a tendency for comfort to increase with effective behaviors. The correlations with challenge were not significant. Resources significantly and negatively correlated with effective acceptance of feedback behaviors. Job satisfaction had a significant positive relationship with effective team spirit and morale behaviors. In general, the total ineffective behaviors negatively correlated with job satisfaction and more strongly and significantly with comfort. As the degree of cooperation increased, the effective giving feedback behaviors decreased. Similarly a negative relationship existed between the value of group interaction and ineffective communication, cooperation, coordination and team spirit and morale in addition to the overall ineffective behaviors. A negative relationship existed between the degree of cooperation rated by external observers and the ineffective cooperation, coordination and overall ineffective behaviors. These negative relationships indicated that the value of the group experience and the perceptions of cooperation decreased as the ineffective behaviors increased. The difficulty of the interaction and overall workload were positively related to cooperation. Furthermore workload was positively related to ineffective communication. These results implied that effective cooperation and ineffective communication imposed additional workloads on the groups. Correlations with Critical Team Behaviors during Design There were negative relationships between mental demand and several effective behaviors including adaptability, communication, coordination, and overall effective behaviors. This suggested as cognitive demands increased there was less attention to group members and group process. In addition, there was a negative relationship between ineffective giving feedback and mental demand. The negative relationships suggested as mental demand increased all types of critical behaviors (effective and ineffective) tended to decrease. As temporal demand increased, the participants tended to show more effective giving and accepting feedback, adaptability, and coordination behaviors although it was offset by ineffective coordination and overall ineffective behaviors. This was probably a response to survive, in that people began to offer or accept help when needed with little attention to the delivery of coordination behaviors. 193 Frustration tended to increase as both ineffective and effective giving and accepting feedback, and ineffective coordination increased. This suggested with additional feedback there was frustration with the work process. Finally the overall NASA TLX ratings indicated positive relationships except for communication, which was negative. All of the correlations between the critical team behaviors and job satisfaction during design indicated slight negative relationships. As comfort decreased, the number of effective acceptance of feedback behaviors increased. Similarly as comfort decreased, there was an increase in ineffective coordination and giving feedback. There was a significant correlation between challenge and effective acceptance behaviors indicating challenge increased as effective behaviors decreased. This relationship with effective acceptance of feedback behaviors also existed for resources and job satisfaction. There was a negative relationship between ineffective giving feedback and job satisfaction. The value of the interaction tended to increase with an increase in effective cooperation behaviors but decrease with effective giving feedback and ineffective team spirit and morale. As the degree of cooperation increased, ineffective cooperation, giving feedback, team spirit and morale and overall ineffective behaviors decreased. Difficulty was positively correlated with ineffective cooperation and giving feedback. Overall workload was positively correlated with effective acceptance of feedback, adaptability, coordination, giving feedback, and overall effective behaviors. In addition there was a positive relationship with ineffective cooperation and giving feedback. External observers scores for difficulty and overall group workload were positively correlated with effective adaptability, giving feedback, and overall effective behaviors and ineffective cooperation, coordination, giving feedback, team spirit and morale and overall ineffective behaviors. Degree of cooperation was excluded from the table because the correlations were very small. Cost effectiveness was negatively correlated with ineffective team spirit and moral while the lifecycle cost was positively correlated with team spirit and morale. System effectiveness increased with effective coordination and ineffective communication. This implied that communication behaviors of any kind were important. Cooperation was believed to be critical for satisfactory project completion (Pinto et al, 1991; Pinto et al., 1993). Correlation coefficients between cooperation and performance were small and insignificant in this study would not support the previous claim. During planning, the correlation between effective cooperation behavior and the Gantt score was negative and not significant (r=-0.13); however the relationship with ineffective cooperation behaviors was negative and significant (r=-0.595, p<0.05). From the overall design perspective, all correlations between the various design performance measures and ineffective cooperation were less than 0.36; correlations for design performance and effective cooperation were less than 0.11, all of which were insignificant. In this study, effective coordination was more highly correlated with design performance using system effectiveness (r=0.717, p<0.01), pointing out the importance for effective coordination in design. 7.6.2 Supplemental Group Process Observations The supplemental data on group process included capturing the number of time-related comments, money-related comments, and non-task related comments. The money and time-comments were included in previous discussions. Planning Project support was significant in the analysis of non-task related comments. While few non-task related comments were made in general, more were observed in treatments with manual support than in treatments with automated support (no comments). Again, this is additional evidence the computer required more attentional resources than planning with manual support. 194 Design Process In general there were few examples of non-task related behaviors or comments that occurred during the design process. Even though there was a paucity of observations, design phase and the interaction between design phase and project support were significant effects. There were significantly more non-task related comments observed in detailed design than in conceptual or preliminary design (about one per every two projects as opposed to one in five or more projects, respectively). This followed the evolution of the design process. As participants completed their assigned task, some offered non-task related comments as a break from their work. Then participants would do one of four things, offer to help and were given new work, offer to help but were not given work, did not offer to help and just watched or did not offer to help and were given unsolicited work to complete. When additional was not given the participant became engaged in non-value added tasks, thus increasing the number of non-task related behaviors. The most non-task related comments occurred in detailed design with automated support (almost one observation per team), which implied that they were under less time pressure than the other groups. The most common reason for the non-task behavior occurred after a participant finished their assignment and had time to talk about non-task related subjects or play with unused LEGOTM pieces. 195 Chapter 8 Conclusions This research confirmed many expected relationships involved in managing an engineering design project. It also identified some factors for fruitful new research. This research provides direction on structuring design projects and future research to develop guidelines on managing engineering design projects. Macroergonomics is concerned with designing organizational and work systems taking into account the pertinent sociotechnical variables and the interactions between these variables (Hendrick, 1986). The macroergonomic perspective began by identifying and analyzing the personnel, technological, and environmental variables and assessing these variables implications on the work systems design. The personnel and technological subsystems within the context of the external environment influenced the organizational structure. Therefore, changes in the organizational structure affected the personnel and technological subsystems. In this study, each subsystem was analyzed to reduce the potential for sub-optimization during the engineering analysis and design process. The resulting experiment was designed and factors within the systems were varied. The dependent variables were selected to represent a balanced set of measures representing the technological and personnel subsystems and the organizational design. Examples of measures were design cycle time, system effectiveness, life-cycle cost, individual mental workload, and group workload. 8.1 Team Design The organizations structural factor was team design. When comparing groups to individuals, individuals performed better than groups on some aspects of performance and on others groups performed better. This was expected. When a division of labor could be introduced to share the tasks among group members, early research indicated groups performed better than individuals (Hare, 1976). Prior research also found that as task complexity increased, groups tended to perform better than individuals due to the ability to share feedback which reduces bias that individuals have unchecked (Davis et al., 1996). Even though the type of feedback may have been delivered in an ineffective manner, there was a positive correlation with reliability (an area where groups tended to perform better than individuals). Regardless of the manner in which feedback was provided, effective or ineffective, it was generally helpful for group performance. The overall implication of the performance results needed careful consideration. As pointed out by others, if an organization is trying to reduce costs, they may be tempted to use the results of studies like this to save money based on the overall cost (Kernan, Bruning, & Miller-Guhde, 1994). When including the planning cost (time and training), which was not accounted for in this study, the overall cost may indeed seem to eliminate project management as a viable option. Furthermore, while overall cost effectiveness was not different for individuals and groups, design costs were 194.7% higher for groups than for individuals. However, when considering the actual reliability of the system designed and some of the intermediate objectives assigned to participants, two issues clearly emerged in favor of using groups. There was clear evidence that groups could attend to some of the important details of design which the individual could not; designs by groups were almost twice as reliable as individuals and they included fewer moving parts, which impacted the maintainability of the system. In addition, the majority of individuals exceeded the cost constraint. The details may have been lost on the individual for one of many reasons including workload, time pressure, or individual bias. The results of this study suggested time pressure was a factor more so than workload, whereas individual bias was not captured. Therefore, this study provided more substantial support for organizations to use groups than the typical explanation of potential effects from satisfaction due to social interactions (Hackman et al., 1975; Hackman, 1987). This study also indicated that at certain points in the design process individuals may be more efficient and effective than groups. 196 In terms of time, groups took 28% longer to plan and 61% longer to complete status report meetings than individuals. Groups taking longer than individuals was a relatively common finding from previous research (for example, Gustafson et al., 1973; Hill, 1982; McGrath, 1984; Barker et al., 1991; Hare, 1992). However, an implication from this study was that during detailed design, individuals took 19% longer than groups. This result supported previous findings that groups were superior to individuals if there was the need for a division of labor (Hare, 1976). During detailed design, while information needed to be shared in order to complete the tasks, the tasks were a series of three different types of activities, thus enabling a division of labor (compared to preliminary design when the activities were more interrelated). The time result and the results for reliability supported Hackman and Morriss (1975) notion that due to large workloads there are times where groups are preferred. That the groups mental demand decreased by 15% from conceptual design to detailed design may be an indication if groups could divide activities into independent tasks their mental workload decreased. The question regarding excessive workload from the comfort facet of job satisfaction, in the analysis within groups supported the perception of workload was significantly stronger during conceptual design (participants agreed workload was reasonable) compared to detailed design (participants slightly agreed). Groups also spent 22% longer in preliminary design than detailed design, indicating that they spent more time selecting the best design for embodiment, which might explain why their reliability scores were higher than the scores for individuals. So when considering these results in total, there was strong support for using groups during the detailed design phase in the engineering design process. As for the social implications of working in groups, there were no differences in job satisfaction. Upon reflecting back over the entire process, individuals had frustration ratings that were 38% high than group ratings. Groups were able verbalize their frustration to other group members and once voiced, may have felt a sense of relief (Yaloms, 1985). These results were consistent with the literature and were discussed relative to previous research in the discussion section. 8.2 Project Support Project support was the independent factor in the technological subsystem. Automation is often suggested as a way to reduce mental demands associated with completing a task (Harris et al., 1995). However, there have been several instances in which the opposite effect occurred as a result of automating (Baeker et al. 1988). These claims had mixed support from this study during planning. Gantt charts created in treatments with automated project support were scored 68% higher than those created with manual support. Planning took 31% longer in treatments with automated support compared to manual support. In addition, the design cost for participants with manual support was 24% higher in treatments without support compared to treatments with automated support, which was directly related to the amount of time spent in the design process and the method with which groups approached design. When considering this issue within groups, there was strong evidence to use automated planning support. The demands based on time pressure and effort were 39% and 33% (respectively) higher for groups using manual project tools than those with automated support. The time pressure was further supported with the question used in the comfort facet of job satisfaction asking if there was sufficient time to complete the task during planning. Groups with automated project support agreed they had sufficient time compared to those with manual project support whose response was between neutral and slightly agree. In the analysis for groups, similar to the analysis comparing individuals to groups, there was mixed support based on the follow up planning questions. Doubt in the ability to complete the design process according to the plan was 55% higher for groups with manual support compared to individuals. Within groups, the automated groups were 38% more confident in the actual plan they developed compared to groups with manual support. Furthermore groups with automated support rated their planning support 27% easier to use and 21% more efficient, and were generally more satisfied with the tools than the groups with manuals project support. 197 It was encouraging that the groups conducting status report meetings did not perceive an additional workload compared to those without status report meetings. This indicated that incorporating status meetings into the design process should not be expected to add to the perception of increased workload. The meetings in this study were conducted without exception at thirty minute time intervals, regardless of what the participants were actually doing in the design process. This was to try to make the situation as realistic as possible in such a controlled environment. The affect of automation on group process stifled group interactions. The number of effective and ineffective critical team behaviors was 83% higher in groups with manual support compared to groups with automated support during planning. For groups, the planning process seemed to have had some initial negative effects on the design process as indicated by workload measured with the NASA TLX, which was 13% higher for groups with project support compared to groups without planning in conceptual design. And job satisfaction was 9% higher for groups that did not plan compared to those that did plan. The group workload scales provided insight based on the participants perceptions of groups during the design process, which changed over time. The value of the group was 33% higher in conceptual design than detailed design. When combined with the time differences in each design phase, this study provides justification for a division of labor during detailed design. The members should work independently - not all tasks should be given to a single person. This research demonstrated individuals working in complete isolation should be avoided. As discussed earlier in Chapter 7, communicating about the activities and coordinating the activities associated with detailed design was important to success. Working in complete isolation would be expected to degrade performance if participants could not communicate changes or new ideas. Because macroergonomics is a human-centered approach to organizational and work system design, improvements 60% 90% in employee job satisfaction and quality measures could be expected (Kleiner & Drury, 1999). In this study, which evaluated the benefits of automating project support, gains of 24%-69% were reported. A main difference for this study compared to previous studies is this study used a controlled laboratory environment and represents a small organizational change. The percentages previously reported were the result of large scale organizational changes. 8.3 Preliminary Guidelines Preliminary guidelines were recommended based on the sociotechnical perspective. The guidelines are a culmination of the effects of varying the organizational structure (team design) and project support (technological subsystem) and determining the impact on performance and the participants completing the design task (personal subsystem). Planning After considering the findings from this study, an overall recommendation is to merge the approaches for the planning process. This study found that the level of automation was the important factor in planning. Based on the group process, groups with manual project support interacted almost twice as much as groups with automated support. To ensure a rich discussion and understanding of the project, the planning should be initially in meetings without automated support. Recall that there were no differences in the quality of the scoping document based on type of tools. Therefore in developing and discussing the information for the scoping document the computer was not important. Using the computer after the meeting is useful to formalize the scoping document. The automated project support enabled Gantt charts that were over 1.5 times better than those produced manually and facilitated changes much more easily than the manual tools. After initially 198 discussing the aspects of the work breakdown structure, time allocations, and resource requirements a software package becomes an important tool in assisting with developing the Gantt chart. The simplicity of the tools and immediate visual feedback suggested that tracking should be conducted with the automated tools. The automated tools may have contributed to some higher perceptions of increased workload. The best solution may be to discuss progress and problems amongst participants and use the computer to share information. Automated tools met the goal of reducing time (by an average of ten minutes in planning) and as a result design cost. To definitively support these recommendations additional research needs to be conducted. Design Projects should be planned and the planning and tracking should be conducted using automated tools to reduce the design cycle. Groups should be employed during the design process to design reliable systems. The tradeoff is in the design costs. With this in mind, conceptual design could be accomplished either individually or in groups, and the performance results would be similar. Structuring the conceptual design process as an individual activity may be a way to introduce a savings in labor cost (resulted in a savings of 1/3). In all cases in this study, all group members were involved in the entire conceptual design phase. And the outcomes were similar between groups and individuals. Time was similar in conceptual design for groups and individuals. Preliminary design should be conducted as a group activity. There was no difference in the time during preliminary design between groups and individuals. Groups did take 22% longer in preliminary design than in detailed design, which may have been better in terms of the design process. Individuals took the same amount of time in preliminary and detailed design. Taking more time in preliminary design, might indicate that a deeper analysis occurred in the comparison of potential solutions. Differences in opinion on the final solution were explored in groups, which may have led to a more thorough analysis compared to that conducted by individuals. Detailed design should be conducted in groups; however the group members should work independently sharing relevant information only when needed. Results of this study showed a need for training on the concept of billable hours. This was in part due to using students as the subjects in this study. The majority did not have work experience from which they could draw on to understand the concept of keeping track the hours spent on a particular project as a basis to charge the project account. 8.4 Future Research The project support tools included in this study were just a few of the tools that can be used to manage projects. In experiments with longer time constraints or in studies on real teams, additional planning and tracking tools can be explored. In more complicated projects, more complex schedule planning techniques can be evaluated to determine if the cycle time can be improved. For example, developing network diagrams and determining the critical path and ways to improve the critical path time can be evaluated. This project in this study had a suggested work breakdown structure for the participants to use because of the fifty-five minute time constraint. Future studies could allow more time for planning which would enable the participants to develop their own work breakdown structures. How automating the planning process affects the creation of a work breakdown structure (WBS) is still unknown. Based on this study, there was evidence that manual project support might lead to a better WBS; but the software would be better for organizing the WBS. As the results of this study indicated, one concern when using the software may discourage the participants from fully engaging in a discussion of the tasks. 199 One of the limitations to generalization in this study was that subjects used the computer on a regular basis. Older populations might not be as familiar with a computer and thus less likely to easily adapt to the software (Nair et al., 1997). To study the impact on users that have the range of experience that can be found on design teams, real teams from industry or an older user population should be studied. In long term projects, the status report meetings can be evaluated for richer data. Data on group process were sparse in this study during status meetings. The meetings were short, lasting one to three minutes. Over the course of a real project, the status meetings take on a more critical role. The meetings are events in which team members make contact to share information that others do not know about. Members often work on various aspects of the project in different work locations. In more realistic meetings the value of automated reporting is expected to be important because the software can be used to show relative progress and identify problems. Studying group process in longer meetings is also important. In addition, determining information needs and the level of detail of the information sharing is important. Training for managing conflict and skills to improve group process are another area for future research. Because this was an initial study, several simplifications were introduced to increase control within the experiment. One example was pencil and paper was used for all design-related activities. Providing the designers and other participants with automated design tools might enhance some of the findings. Use of multiple computer tools might lead to a new awareness on the impact of automating and differentiate between when planning and tracking should be conducted via computer or manually. The suspicion is that the combined automation of design and planning will accumulated to impact workload more than what was found in this study. In addition, the critical team behaviors are expected to have an even larger difference between manual and automated treatments because of the increased attention directed to the computer. Another controlled experiment may be useful to evaluate some of recommendations made in the guidelines. In a future experiment, the group structure should be a factor. The structure should be varied between the guidelines recommended in this study and a control, in which the entire process was conducted collaboratively without independent work. In addition, a longer study over the course of several design periods should be employed (similar to the study conducted by Harvey, 1987) to simulate a more realistic design experience. Therefore, the initial effects of planning would not immediately impact conceptual design. This type of an experiment; while risky because participants might more readily drop out, would provide a better simulation of a true design experience. Conducting an experiment over a longer time period in a more realistic setting would provide a more realistic scenario with longitudinal results. One example might be to use a senior design capstone course. Conducting the study of over several months as part of a course provides the participants a problem with real consequences. A more realistic pattern of problems would occur that did not have the opportunity to emerge in a two to three-hour experiment. Real scheduling problems and communication breakdowns are just a two examples of the problems that could emerge. The impact of leadership may be a valuable topic for future research. Several different types of leadership would be relevant to explore. The first is emergent leadership, which is when individuals respond to a person that they perceive to be the most influential group member (Pearce & Conger, 2003). Emergent leadership can only occur after the group has spent time together and had the opportunity to communicate, for example seeking the opinion of others, being informed about what other group members are doing, initiating new ideas and being flexible to try new ideas. Anecdotally, in this study, some groups appeared to have an emergent leader. Another form of leadership to explore in relation to engineering design is situational leadership. Under situational leadership, different group members have stronger abilities in different areas and they influence the group when their skills are called for (Northhouse, 2004). A third type of leadership to explore is designated leadership (Northhouse, 2004). In this study project leaders were not assigned. However, in some groups the project was initiated only 200 after a brief discussion of who would be the leader. Building on the work of Northhouse (2004) and Pearce and Conger (2003), leadership of engineering design teams could be a productive topic to investigate. Several of the outcomes indicated that individuals might perform better than the groups, specifically in conceptual design, and possibly in planning (since performance was not affected by team design). The concern with both planning and conducting conceptual design as an individual activity is that in the subsequent stages, there may not be buy-in to the work process nor the design concepts. To determine whether this violates the intentions of participatory design and sociotechnical systems design guidelines, additional research needs to be conducted. A slight variation on the current study is to include the interdependencies among group members as an independent factor. For example, during conceptual design the work was collaborative with the group members working together to generate ideas and identify design requirements. In preliminary design, some aspects of the phase were collaborative, while other aspects were independent. And in detailed design the work was primarily independent. Evaluating group process, group workload, mental workload, and job satisfaction in these different types of dependencies may lead to additional recommendations for structuring design projects. The role that participants played on design teams is another issue for further study. In this study, the roles were forced through random assignment. However, an observation occurred in several of the trials based on gender (while this was not a factor studied it does raise questions for future research). Several females assigned to the design role, reassigned the drawing responsibility to another group member. Males, without exception, did not reassign their assigned role to another member. The interesting question here is why? This finding may be related to self-efficacy in design. All of the participants passed the detailed drawing training activity. Teams are commonly used in engineering education. The general model used here for studying an engineering design process would seem to be applicable to studying important facets of the engineering educational model. The important interrelationships exist in both models and need to function well for success. Based on this study, some groups seemed to collaborate at certain points, where as other groups attempted to accomplish each task together until time became an issue. When members were dysfunctional, it was interesting to see how group members dealt with and accommodated the troublesome members. Therefore the use of teams as a tool in the academic setting raises questions such as: How useful are teams?; What are the conditions within which teams are justified?; and What type of projects led to worthy educational experiences? 8.5 Practical Implications The following are the practical implications from this study: Team Design Benefits of groups o Attended to important details of design Spent longer in preliminary than detailed design Higher reliability Better able to remain within cost constraints o Less frustrated upon reflection than individuals Drawbacks of groups o Higher design costs o Took longer to plan and complete status meetings 201 Determining whether or not to use groups or individuals must be a function of the organizational goals and what is the most important performance measure. For the design task, there was a trade off between the reliability of the system designed and the cost of that system as a result of the team design. Project Support Benefits of automation in planning o Better Gantt charts o Completed more quickly o Lower design costs o Lower perceptions of time pressure and effort Drawbacks of automation in planning o Stifled group interaction Drawbacks of planning & tracking o Initial negative effects within groups in design based on the NASA TLX o Doubt in the design was lower within groups with project support o Job satisfaction was lower for groups that planned compared to groups that did not Evidence for automated planning and tracking within groups o More secure with the plan developed with automated support o Rated the automated support as easier to use and more efficient o Users with automated project support were more satisfied with their support than those with manual project support Because automating the project support in this project meet the goals for automation: reducing time and cost, this study provides support for automating the support. The following are a summary of the recommendations for managing future engineering design projects: Planning Design Conceptual design: Individual or Group o No difference in performance between individuals and groups o Using individuals will save money Preliminary design: Group o Time elapsed similar between groups and individuals o Groups took longer in preliminary design than detailed design o Groups were able to attend to more design requirements Detailed design: Group o Groups took less time than individuals o Group members should work independently, sharing relevant information when needed Train on billable hours Integrate status report meetings with automated support to reinforce awareness of time and money Use pencil & paper early in planning Formalize planning information and scheduling with software For tracking, discuss progress and problems between group members Use computer to share information 202 References Aasman, J., Mulder, G., & Mulder, L. (1987). Operator Effort and the measurement of heart-rate variability. Human Factors, 29(2), 161-170. Adachi, T., Shih, L.C., & Enkawa, T. (1994). Strategy for supporting organization and structuring of development teams in concurrent engineering. The International Journal of Human Factors in Manufacturing, 4(2), 101-120. Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B.N. Petrov & F. Csaki (Eds.) 2nd International symposium on information theory (pp. 267-281). Akademia Kiado, Budapest. Akerstedt, T., & Gillberg, M. (1990). Subjective and objective sleepiness in the active individual. International Journal of Neuroscience, 52, 29-37. Altman, I. (1966). Aspects of the criterion problem in small group research, I. Behavioral domains to be studied. Acta Psychologica, 25, 101-131. Appley, M.H., & Trumbull, R. (1986). Development of the stress concept. In M.H. Appley & R. Trumbull (Eds.), Dynamics of stress. New York: Plenum Press. Archibald, R.D. (1976). Managing high-technology programs and projects. New York: John Wiley & Sons. Argyle, M. (1992). The social psychology of everyday life. London: Routledge. Asimow, A. (1962). Introduction to design. Englewood Cliffs, NJ: Prentice Hall. Astrand, P., & Rohdal, K. (1986). Textbook of work physiology. New York: McGraw-Hill. Baecker, R.M. (1993). Introduction. In R.M. Baecker (Ed.), Readings in groupware and computersupported cooperative work. San Mateo, CA: Morgan Kaufmann. Baeker, R.M., & Buxton, W.A.S. (1988). Readings in HCI: A multidisciplinary approach. Los Altos, CA: Morgan Kaufmann. Bailey, J.M., & Bhagat, R.S. (1987). Meaning and measurement of stressors in the work environment: An evaluation. In S.V. Kaskl & C.L. Cooper (Eds.), Stress and health: issues in research methodology. New York: John Wiley & Sons. Baker, K., Ball, L., Culverhouse, P., Dennis, I., Evans, J., Jagodzinski, P., Pearce, P., Scothern, D., & Venner, G. (1991). A psychologically based intelligent design aid. In P.J.W. ten Hagen and P.J. Veerkamp (Eds.), Intelligent CAD systems III. New York: Springer-Verlag. Baker, D.P., & Salas, E. (1992). Principles for measuring teamwork skills. Human Factors, 34(4), 469475. Bales, R.F. (1950). Interaction process analysis. Chicago: University of Chicago Press. Ball, L.J., Evans, J.St.B.T., & Dennis, I. (1994). Cognitive processes in engineering design: a longitudinal study. Ergonomics, 37(11), 1753-1786. Banderet, L.E., Stokes, J.W., Francesconi, R., & Kowal, D. (1980). Simulated, sustained-combat operations in the field artillery fire direction center (FDC): A model for evaluating biomedical indices. In Proceedings of the 1980 army science conference, Vol. 1. West Point, NY: U.S. Military Academy. 203 Banderet, L.E., Stokes, J.W., Francesconi, R., Kowal, D.M., & Naitoh, P. (1981). Artillery teams in simulated sustained combat: performance and other measures. In L.C. Johnson, W.P. Colquhoun, D.I. Tepas, & M.J. Colligan (Eds.) Biological rhythms, sleep and shift work. Jamaica, NY: Spectrum. Barker, L.L., Wahlers, K.J., Watson, K.W., & Kibler, R.J. (1991). Groups in process: an introduction to small group communication. Englewood Cliffs, NJ: Prentice Hall. Beekun, R.I. (1989). Assessing the effectiveness of sociotechnical interventions: Antidote or fad? Human Relations, 42(10), 877-897. Beith, B.H. (1987). Subjective workload under individual and team performance conditions. In Proceedings of the human factors society 31st annual meeting (pp. 67-71). Santa Monica, CA: Human Factors Society. Berkowitz, L. (1954). Group standards, cohesiveness, and productivity. Human Relations, 7, 509-519. Bertodo, I.R. (1994). Engineering design: Academic dilemma. International Journal of Vehicle Design, 15(6), 545-562. Bietz, W., Langer, T., Luczak, H., Mueller, T., & Springer, J. (1990). Evaluation of a compact CAD course in laboratory experiments. International Journal of Human Computer Interaction, 2(2), 111-135. Blanchard, B.S., & Fabrycky, W.J. (1990). Systems engineering and analysis, Englewood Cliffs, NJ: Prentice Hall. Blickensderfer, E., Salas, E., & Cannon-Bowers, J. (2000). When the teams came marching home. In M.M. Beyerlein (Ed.), Work teams: past, present, and future. Dordrecht: Kluwer Academic Publishers. Borg, G. (1971). Psychological and physiological studies of physical work. In W.T. Singleton, J.G. Fox, & D. Whitfield (Eds.), Measurement of man at work. London: Taylor and Francis. Borg, G. (1982). Psychological bases of perceived exertion. Medicine and Science in Sports and Exercise, 14(5), 377-381. Borg, G. (1998). Borgs perceived exertion and pain scales. Champaign, IL: Human Kinetics. Bowers, C.A., Braun, C.C., & Morgan, Jr., B.B. (1997). Team workload: its meaning and measurement. In M.T. Brannick, E. Salas, & C. Prince (Eds.), Team performance assessment and measurement (pp.85-108). Mahwah, NJ: Lawrence Erlbaum. Bowers, C.A., Salas, E., Prince, C., & Brannick, M. (1992). Games team play: a method for investigating team coordination and performance. Behavior Research Methods, Instruments, & Computers 24(4), 503-506. Bowers, C.A., Urban, J.M., & Morgan, Jr., B.B. (1992). The study of crew coordination and performance in hierarchical team decision making (Report No. TR-92-01). Orlando, FL: University of Central Florida, Team Performance Laboratory. Brannick, M.T., Prince, A., Prince, C., & Salas, E. (1995). The measurement of team process. Human Factors, 37(3), 641-651. Brannick, M.T., Roach, R.M., & Salas, E. (1993). Understanding team performance: a multimethod study. Human Performance, 6(4), 287-308. Brett, J.M., & Rognes, J.K. (1986). Intergroup relations in organizations. In P.S. Goodman (Ed.), Designing effective work groups. San Francisco: Joey-Bass. 204 Briggs, G.E., & Naylor, J.C. (1965). Team versus individual training, training task fidelity, and task organization effects on transfer performance by three-man teams. Journal of Applied Psychology, 49(6), 387-392. Burke, K., Aytes, K., Chidambaram, L., & Johnson, J.J. (1999). A study of partially distributed work groups: The impact of media, location, and time on perception and performance. Small Group Research, 30(4), 453-490. Burke, K., & Chidambaram, L. (1994). Development in electronically-supported groups: A preliminary longitudinal study of distributed and face-to-face meetings. In Proceedings of the 27th annual Hawaii international conference on system sciences (pp. 104-113). Piscataway, NJ: IEEE. Burns, T., & Stalker, G.M. (1961). The management of innovation. London: Tavistock. Bursic, K.M. (1992). Strategies and benefits of the successful use of teams in manufacturing organizations. In IEEE transactions on engineering management, 39(3), (pp. 277-289). Piscataway, NJ: IEEE Cammann, C., Fichman, M., Jenkins, D., & Klesh, J. (1979). The Michigan organizational assessment questionnaire. Unpublished manuscript, University of Michigan, Ann Arbor, MI. Campbell, J.P. (1968). Individual versus group problem solving in an industrial sample. Journal of Applied Psychology, 42(3), 205-210. Campbell, D.T., & Fiske, D.W. (1959). Convergent and discriminant validation by the multitraitmultimethod matrix. Psychological Bulletin, 56, 81-105. Campion, M.A., & Medsker, G.J. (1992). Job design. In G. Salvendy (Ed.), Handbook of industrial engineering (pp. 845-881). New York: John Wiley & Sons. Cannon-Bowers, J.A., Tannenbaum, S.I., Salas, E., & Volpe, C.E. (1995). Defining competencies and establishing team training requirements. In R.A. Guzzo & E. Salas & Associates (Eds.) Team effectiveness and decision making in organizations. San Francisco: Jossey-Bass. Cano, A., & Kleiner, B.M. (1996). Sociotechnical design factors for virtual team performance. In Proceedings of the human factors and ergonomics society 40th annual meeting (pp. 786-790). Santa Monica, CA: Human Factors and Ergonomics Society. Cano, A.R., Meredith, J.W., & Kleiner, B.M. (1998). Distributed and collocated group communication vs. decision systems. In P. Vink, E.A.P. Koningsveld, & S. Dhondt (Eds.), Human factors in organizational design and management Vol. 6 (pp. 501-506). North-Holland: Elsevier. Carayon-Sainfort, P. (1992). The use of computers in offices: impact on task characteristics and worker stress. International Journal of Human-Computer Interaction, 4(3), 245-261. Cartwright, D. (1968). The nature of group cohesiveness. In D. Cartwright & A. Zander (Eds.), Group dynamics: Research and theory. New York: Harper & Row. Casali, J.G., & Wierwille, W.W. (1983). A comparison of rating scale, secondary-task, physiological, and primary-task workload estimation techniques in a simulated flight task emphasizing communications load. Human Factors, 25(6), 623-641. Cherns, A. (1987). Principles of sociotechnical design revisited. Human Relations, 40(3), 153-162. Cherns, A.B. (1976). The principles of sociotechnical design. Human Relations, 9(8), 783-792. Cook, J.D., Hepworth, S.J., Wall, T.D., & Warr, P.B. (1981). The experience of work: a compendium and review of 249 measures and their use. London: Academic Press. 205 Cornell, P., Luchetti, R., Mack, L.A., & Olson, G.M. (1989). CSCW anecdotes and directions. In Proceedings of the human factors society 33rd annual meeting (pp. 867-870). Santa Monica, CA: Human Factors Society. Cummings, T.G., Molloy, E.S., & Glen, R. (1977). A methodological critique of fifty-eight selected work experiments. Human Relations, 30(8), 675-708. Curran, S.L., Andrykowski, M.A., & Studts, J.L. (1995). Short form of the profile of mood states (POMSSF): psychometric information. Psychological Assessment, 7(1), 80-83. Dailey, R.C. (1980). A path analysis of R&D team coordination and performance. Decision Sciences, 11, 357-369. Daft, R.L., & Lengel, R.H. (1984). Information richness: a new approach to managerial behavior and organization design. Research in Organizational Behavior, 6, 191-233. Daft, R.L., & Lengel, R.H. (1986). Organizational information requirements, media richness and structural design. Management Science, 32(5), 554-571. Davis, D.D., & Harless, D.W. (1996). Group vs. individual performance in a price-searching experiment. Organizational Behavior and Human Decision Processes, 66(2), 215-227. Davis, L.T., Gaddy, C.D., & Turney, J.R. (1985). An approach to team skills training of nuclear power plant control room crews (NUREG/CR-4255GP-R-123022). Columbia, MD: General Physics Corp. Davis, L.V., & Toseland, R.W. (1987). Group versus individual decision making: an experimental analysis. Social Work with Groups, 10(2), 95-110. Deeb, J.M., & Drury, C.G. (1992). Perceived exertion in isometric muscular contractions related to age, muscle, force level and duration. In Proceedings of the human factors society 36th annual meeting (pp. 712-716). Santa Monica, CA: Human Factors Society. DeGreene, K.B. (1973). Sociotechnical systems: factors in analysis, design, and management. Englewood Cliffs, NJ: Prentice-Hall. Dennis, A.R., Nunamaker, J.F., & Vogel, D.R. (1990). A comparison of laboratory and field research in the study of electronic meeting systems. Journal of Management Information Systems, 7(3), 107135. Dieter, G.E. (1991). Engineering design a materials and processing approach. New York: McGraw-Hill. Dinges, D.F., & Graeber, R.C. (1989). Crew fatigue monitoring. Flight Safety Digest, October, 65-75. Dockery, C. A., & Neuman, T. (1994). Developing a better design process: redesigning the organization to produce more ergonomic designs. In Proceedings of the human factors and ergonomics society 38th annual meeting (pp.759-763). Santa Monica, CA: Human Factors and Ergonomics Society. Driskell, J.E., & Salas, E. (1991). Group decision making under stress. Journal of Applied Psychology, 76(3), 473-478. Driskell, J.E., & Salas, E. (1992). Collective behavior and team performance. Human Factors, 34(3), 277288. Dubrovsky, V.J., Kiesler, S., & Sethna, B.N. (1991). The equalization phenomenon: status effects in computer-mediated and face-to-face decision-making groups. Human-Computer Interaction, 6(2), 119-146. Dyer, J.L. (1984). Team research and team training: a state-of-the-art review. In F.A. Muckler (Ed.), Human factors review (pp. 285-323). Santa Monica, CA: Human Factors Society. 206 Dym, C.L. (1994). Engineering design: A synthesis of views. Cambridge: Cambridge University Press. Eggemeier, F.T. (1988). Properties of workload assessment techniques. In P.A. Hancock & N. Meshkati (Eds.), Human mental workload. North Holland: Elsevier. Ehrlenspiel, K., & Dylla, N. (1993). Experimental investigation of designers thinking methods and design procedures. Journal of Engineering Design, 4(3), 201-212. Elliot, G.R., & Eisdorfer, C. (1982). Stress and human health. New York: Springer. Evans, S. (1990). Implementation framework for integrated design teams. Journal of Engineering Design, 1(4), 355-363. Everly, G.S., & Soelmann, S.A. (1987). Assessment of the human stress response. New York: AMS Press. Fernstrom, E., & Ericson, M.O. (1997). Computer mouse or trackpoint effects on muscular load and operator experience. Applied Ergonomics, 28(5/6), 347-354. Ferguson, E.S. (1992). In the minds eye. Cambridge, MA: MIT Press. Finkelman, J.M. (1994). A large database study of the factors associated with work-induced fatigue. Human Factors, 36(2), 232-243. Fitts, P.M. (1951). Engineering psychology and equipment design. In. S.S. Stevens (Ed.), Handbook of experimental psychology (pp. 1287-1340). New York: John Wiley and Sons. Fleishman, E.A., & Zaccaro, S.J. (1992). Toward a taxonomy of team performance functions. In R.W. Swezey & E. Salas (Eds.), Teams: their training and performance. Norwood, NJ: ABLEX. Foushee, H.C. (1984). Dyads and triads at 35,000 feet: factors affecting group process and aircrew performance. American Psychologist, 39(8), 885-893. Fox, T.L., & Spence, J.W. (1998). Tools of the trade: a survey of project management tools. Project Management Journal, 29(3), 20-27. Furnham, A., & Yazdanpanahi, T. (1995). Personality differences and group versus individual brainstorming. Personality and Individual Differences, 19(1), 73-80. Frankenberger, E., & Auer, P. (1997). Standardized observation of team-work in design. Research in Engineering Design, 9, 1-9. Fried, Y, Rowland, K.M., & Ferris, G.R. (1984). The physiological measurement of work stress: a critique. Personnel Psychology, 37, 583-615. Fulk, J., Steinfield, C.W., Schmitz, J., & Power, J.G. (1987). A social information processing model of media use in organization. Communications Research, 14(5), 529-552. Gaillard, A.W.K. (1993). Comparing the concepts of mental load and stress. Ergonomics, 36(9), 9911005. Gay, G., & Lentini, M. (1995). Use of communication resources in a networked collaborative design environment. The Journal of Computer Mediated Communication, 1(1). (http://www.ascusc.org/jcmc/jcmc.html) Gersick, C. (1988). Time and transition in work teams: toward a new model of group development. Academy of Management Journal, 31(1), 9-41. Gibson, J.L., Ivancevich, J.M., & Donnelly, J.H. (1997). Organizations, behavior structure processes. Boston: McGraw-Hill. Gido, J., & Clements, J.P. (1999). Successful project management. Cincinnati, OH: South-Western College Publishing. 207 Gillberg, M., Kecklund, G., & Akerstedt, T. (1994). Relations between performance and subjective ratings of sleepiness during a night awake. Sleep, 17(3), 236-241. Gist, M.E., Locke, E.A., & Taylor, M.S. (1987). Organizational behavior: Group structure, process, and effectiveness. Journal of Management, 13(2), 237-257. Glickman, A.S., Zimmer, S., Montero, R.C., Guerette, P.J., Campbell, W.J., Morgan, B.B., & Salas, E. (1987). The evolution of teamwork skills: An empirical assessment with implications for training. (Tech. Report 87-016). Orlando, FL: Naval Training Systems Center. Goel, V., & Pirolli, P. (1989). Motivating the notion of generic design within information-processing theory: the design problem space. AI Magazine, 10(1), 18-38. Goodacre, D.M. (1953). Group characteristics of good and poor performing combat units. Sociometry, 16(2), 168-179. Gopher, D., & Donchin, E. (1986). Workload an examination of the concept. In K.R. Boff, L. Kaufman, & J.P. Thomas (Eds.), Handbook of perception and human performance Vol. 2. New York: Wiley. Grace, D.R., & Billatos, S.B. (1992). Application of concurrent engineering in the redesign process. In Design for manufacture (pp. 23-27). New York: ASME. Graetz, K.A., Boyle, E.S., Kimble, C.E., Thompson, P., & Garloch, J.E. (1998). Information sharing in face-to-face, teleconferencing, and electronic groups. Small Group Research, 29(6), 714-743. Greiner, R., & Metes, G. (1995). Going virtual. Upper Saddle River, NJ: Prentice Hall. Grise, M.L. (1997). Information overload in face-to-face electronic meetings. Unpublished doctoral dissertation, Queens University, Kingston Ontario, Canada. Gully, S. (2000). Work teams research. In M.M. Beyerlein (Ed.), Work teams: Past, present, and future. Dordrecht: Kluwer Academic Press. Gustafson, D.H., Shukla, R.K., Delbecq, A., & Walster, G.W. (1973). A comparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups. Organizational Behavior and Human Performance, 9(2), 280-291. Hackman, J.R. (1987). The design of work teams. In J.W. Lorsch (Ed.), Handbook of organizational behavior (pp. 315-342). Englewood Cliffs, NJ: Prentice Hall. Hackman, J.R., Brousseau, K.R., & Weiss, J.A. (1976). The interaction of task design and group performance strategies in determining group effectiveness. Organizational Behavior and Human Performance, 16, 350-365. Hackman, J.R., & Morris, C.G. (1975). Group tasks, group interaction process, and group performance effectiveness: a review and proposed integration. In L. Berkowitz (Ed.), Advances in experimental social psychology (pp. 45-99). New York: Academic Press. Hackman, J.R., & Oldham, G.R. (1975). Development of the job diagnostic survey. Journal of Applied Psychology, 60(2), 159-170. Hackman, J.R., & Vidmar, N. (1970). Effects of size and task type on group performance and member reactions. Sociometry, 33, 37-54. Hall, E.R., & Rizzo, W.A. (1975). An assessment of U.S. navy tactical team training (TAEG Report 18). Orlando, FL: Training Analysis and Evaluation Group. 208 Hankins, T.C., & Wilson, G.F. (1998). A comparison of heart rate, eye activity, EEG and subjective measures of pilot mental workload during flight. Aviation Space and Environmental Medicine, 69(4), 360-367. Hanna, D.P. (1988). Designing organizations for high performance, Reading, MA: Addison-Wesley. Hare, A.P. (1976). Handbook of small group research. New York: Free Press. Hare, A.P. (1981). Group size. American Behavioral Scientist, 24(5), 695-708. Hare, A.P. (1992). Groups, teams, and social interaction. New York: Praeger. Harmer, S. (1998). Multi-crew workload measurement for Nimrod MRA4. In Collaborative crew performance in complex operational systems. Hull, Canada: RTO/NATO. Harris, W.C., Goernert, P.N., Trench, B., & Sachau, D. (1995). Predicting performance variability: The accuracy of vigor and fatigue self-reports. In Proceedings of the human factors and ergonomics society 39th annual meeting (pp. 922-925). Santa Monica, CA: Human Factors and Ergonomics Society. Harris, W.C., Hancock, P.A., Arthur, E.J., & Caird, J.K. (1995). Performance, workload, and fatigue changes associated with automation. The International Journal of Aviation Psychology, 5(2), 169185. Hart, S.G., & Staveland, L.E. (1988). Development of NASA-TLX (task load index): Results of empirical and theoretical research. In P.A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 139-183). North-Holland: Elsevier. Hart, S.G., Hauser, J.R., & Lester, P.T. (1984). Inflight evaluation of four measures of pilot workload. In Proceedings of the human factors 28th annual meeting (pp. 945-949). Santa Monica, CA: Human Factors Society. Harvey, C.M. (1997). Toward a model of distributed engineering collaboration. Unpublished doctoral dissertation, Purdue University. Harvey, C.M., & Koubek, R.J. (1998). Toward a model of distributed engineering collaboration. In Computers and industrial engineering, 35(1-2) (pp. 173-176). Great Britain: Elsevier. Helmreich, R.L. (1984). Cockpit management attitudes. Human Factors, 26(5), 583-589. Hendrick, H. (1986). Macroergonomics: a conceptual model for integrating human factors with organizational design. In Human factors in organizational design and management - II (pp. 467477). North-Holland: Elsevier. Hendrick, H.W. (1997). Organizational design and macroergonomics. In G. Salvendy (Ed.), Handbook of human factors and ergonomics. New York: John Wiley. Hendrick, H.W., & Kleiner, B.M. (2000). Macroergonomics: an introduction to work system design. In Proceedings of the human factors and ergonomics society 44th annual meeting. Santa Monica, CA: Human Factors and Ergonomics Society. Herzberg, F. (1966). Work and the nature of man. Cleveland, OH: World Publishing. Hicks, T.G., & Wierwille, W.W. (1979). Comparison of five mental workload assessment procedures in a moving-base driving simulator. Human Factors, 21(2), 129-143. Hill, G.W. (1982). Group versus individual performance: Are N + 1 heads better than one? Psychological Bulletin, 91(3), 517-539. 209 Hiltz, S.R., Johnson, K., and Turoff, M. (1986). Experiments in group decision making communication process and outcome in face-to-face versus computerized conferences. Human Communication Research, 13(12), 225-252. Hiltz, S.R., Turoff, M., and Johnson, K. (1981). The effect of structure, task, and individual attributes on consensus in computerized conferences. In Proceedings of the IFIP TC-6 international symposium on computer message systems (pp. 91-102). Amsterdam: North Holland. Hinkelmann K., & Kempthorne, O. (1994). Design and analysis of experiments. New York: Wiley. Hinton, J.W., Rotheiler, E., & Howard, A. (1991). Confusion between stress and state anxiety in a much used self-report stress inventory. Personality and Individual Differences, 21(1), 91-94. Hobfoll, S.E. (1989). Conservation of resources, a new attempt at conceptualizing stress. American Psychologist, 44 (3), 513-524. Hobfoll, S.E., & Lilly, R.S. (1993). Resource conservation as a strategy for community psychology. Journal of Community Psychology, 21, 128-148. Hockey, G.R.J., Briner, R.B., Tattersall, A.J., & Wiethoff, M. (1989). Assessing the impact of computer workload on operator stress: The role of system controllability. Ergonomics, 32(11), 1401-1418. Hoffman, L.R. (1979). Applying experimental research on group problem solving to organizations. Journal of Applied Behavioral Science, 15, 375-391. Hogan, J., Neubauer, R.M., & Salas, E. (1989). Deriving performance measures for team tasks: Evaluating a methodology. In Proceedings of the human factors society 33rd annual meeting (pp. 1273-1277). Santa Monica, CA: Human Factors Society. Hollnagel, E., & Bye, A. (2000). Principles for modeling function allocation. International Journal of Human-Computer Studies, 52, 253-265. Holloman, C.R., & Hendrick, H.W. (1971). Problem solving in different sized groups. Personnel Psychology, 24, 489-500. Holt, K. (1993). Computer aided creativity in engineering design. Journal of Engineering Design, 4(4), 371-376. Honaker, R.E. (1996). Assessing trailer material handling tasks: biomechanical modeling, posture categorization, physiological measure, and subjective rating. Unpublished thesis, Virginia Polytechnic Institute and State University, Blacksburg, VA. Hoppe, R.A. (1967). Analysis, synthesis, and solutions of the doodlebug problem by groups and individuals. Psychological Reports, 20, 1139-1145. Howell, D.C. (1992). Statistical methods for psychology 3rd ed. Belmont, CA: Duxbury Press. Huey, B.M., & Wickens, C.D. (1993). Workload transition: Implications for individual and team performance. Washington, D.C.: National Academy Press. Iaffaldano, M.T., & Muchinsky, P.M. (1985). Job satisfaction and job performance. Psychological Bulletin, 97(2), 251-273. Ilgen, D.R., Shapiro, J., Salas, E., & Weiss, H. (1987). Functions of group goals: Possible generalizations from individuals to groups (Tech. Report 87-022). Orlando, FL: Human Factors Division, Naval Training Systems Center. Imada, A.S., & Bowman, M.E. (1992). Change by design, human factors by necessity: lessons in participatory ergonomics. In Proceedings of the human factors society 36th annual meeting (pp. 855-858). Santa Monica, CA: Human Factors Society. 210 Jewell, L.N. (1998). Contemporary industrial/organizational psychology. Pacific Grove: Brooks/Cole Publishing. Jex, S.M., & Gudanowski, D.M. (1992). Efficacy beliefs and work stress: an exploratory study. Journal of Organizational Behavior, 13, 509-517. Johannsen, G., Levis, A.H., & Stassen, H.G. (1994). Theoretical problems in man-machine systems and their experimental validation. Automatica, 30(2), 217-231. Johansson, G., & Aronsson, G. (1984). Stress reactions in computerized administrative work. Journal of Occupational Behavior, 5, 159-181. Johansson, G. (1984). Computer technology: Stress and health relevant transformations of psychosocial work environments. In G. Salvendy, (Ed.), Human-computer interaction (pp. 347-353). Amsterdam: Elsevier. Johnson, DA., & Weiss, D.J. (1971). Middle management decision making and job satisfaction. Danville, IL: Interstate Printers and Publishers. Johnson, H.H., & Torcivia, J.M. (1967). Group and individual performance on a single-stage task as a function of distribution of individual performance. Journal of Experimental Social Psychology, 3(3), 266-273. Johnston, W.A. (1966) Transfer of team skills as a function of type of training. Journal of Applied Psychology, 50(2), 102-108. Johnston, W.A., & Briggs, G.E. (1968). Team performance as a function of team arrangement and work load. Journal of Applied Psychology, 52(2), 89-94. Jorna, P.G.A.M. (1992). Spectral analysis of heart rate and psychological state: A review of its validity as a workload index. Biological Psychology, 34(2-3), 237-257. Jordan, B., & Henderson, A. (1995). Interaction analysis: Foundations and practice. The Journal of the Learning Sciences, 4(1), 39-103. Kale, W.L., & Stenmark, D.E. (1983). A comparison of four life event scales. American Journal of Community Psychology, 11(4), 441-458). Kalimo, R., Harju, A., Leskinen, J., & Nykyri, E. (1992). Work load, psychophysiological strain, and coping of innovation project management in industry. Scandinavian Journal of Work and Environmental Health, 18(supplement 2), 130-132. Kalimo, R., Lindstrom, L., & Smith, M. (1997). Psychosocial approach in occupational health. In G. Salvendy (Ed.), Handbook of human factors and ergonomics. New York: John Wiley. Kanki, B.G. (1996). Stress and aircrew performance: a team-level perspective. In J.E. Driskell & E. Salas (Eds.), Stress and human performance. Mahwah, NJ: Lawrence Erlbaum. Kantowitz, B.H., & Sorkin, R.D. (1987). Allocation of functions. In G. Salvendy (Ed.), Handbook of human factors (pp. 355-369). New York: John Wiley and Sons. Kayworth, T.D., & Leidner, D. (2000). The global virtual manager: a prescription for success. European Management Journal, 18(2), 183-194. Kernaghan, J.A., & Cooke, R.A. (1986). The contribution of the group process to successful project planning in R&D settings. IEEE Transactions on Engineering Management, 33(3), 134-140. Kernaghan, J.A., & Cooke, R.A. (1990). Teamwork in planning innovative projects: improving group performance by rational and interpersonal interventions in group process. IEEE Transactions on Engineering Management, 37(2), 109-116. 211 Kernan, M.C., Bruning, N.S., & Guhde, L.M. (1994). Individual and group performance: Effects of task complexity and information. Human Performance, 7(4), 273-289. Kidd, J.S. (1961). A comparison of one-, two-, and three-man work units under various conditions of workload. Journal of Applied Psychology, 45(3), 195-200. Kidd, P.T. (1994). Agile manufacturing: Forging new frontiers. New York: Addison-Wesley. Kies, J.K. (1997). Empricial methods for evaluating video-mediated collaborative work. Unpublished doctoral dissertation. Virginia Polytechnic Institute and State University, Blacksburg, VA. Kies, J.K., Williges, R.C., & Rosson, M.B. (1997). Evaluating desktop video conferencing for distance learning applications. Computers and Education, 28(2), 79-91. Kilbom, A. (1990). Measurement and assessment of dynamic work. In J.R. Wilson & E.N. Corlett (Eds.), Evaluation of human work. London: Taylor and Francis. King, M.G., Burrows, G.D., & Stanley, G.V. (1983). Measurement of stress and arousal: Validation of the stress/arousal checklist. British Journal of Psychology, 74, 473-479. Kinney, S., & Dennis, A. (1994). Reevaluating media richness: Cues, feedback, and task. In Proceedings of the 27th annual Hawaii international conference on systems sciences (pp. 21-30). Piscataway, NJ: IEEE. Kliem, R.L. (1986). The secrets of successful project management. New York: John Wiley & Sons. Kliem, R.L., & Ludin, I.S. (1998). Project management practitioners handbook. New York: Amacom. Klein, T.J., & Cassidy, W.B. (1972). Relating operator capabilities to system demands. In Proceedings of the human factors society 16th annual meeting (pp. 324-334). Santa Monica, CA: Human Factors Society. Kleiner, B.M. (1996). Macroergonomic lessons learned from large scale change efforts in industry, government, and academia. In O. Brown, & H.W. Hendrick (Eds.), Human factors in organizational design and management V (pp. 483-488). Amsterdam: Elsevier. Kleiner, B.M. (1998). Macroergonomic directions in function allocation. In P. Vink, E.A.P., Koningsveld, & Dhondt (Eds.), Human factors in organizational design and management VI (pp. 635-640). Amsterdam: Elsevier. Kleiner, B.M., & Drury, C.G. (1999). Large-scale regional economic development: macroergonomics in theory and practice. Human Factors and Ergonomics in Manufacturing, 9(2), 151-163. Kleinman, D.L. (1990). Coordination in human teams: theories, data and models. In U. Jaaksoo & V.I. Utkin (Eds.), IFAC 11th Triennial World Congress Vol. 5 (pp. 417-422). Oxford: Pergamon Press. Knutson, J., & Bitz, I. (1991). Project management: How to plan and manage successful projects. New York: Amacom. Kobasa, S.C., Maddi, S.R., & Courington, S. (1981). Personality and constitution as mediators in the stress-illness relationship. Journal of Health and Social Behavior, 22, 368-378. Kohlisch, O., & Schaefer, F. (1996). Physiological changes during computer tasks: Responses to mental load or to motor demands? Ergonomics, 39(2), 213-224 Kramer, A.F. (1991). Physiological metrics of mental workload. In D.L. Damos (Ed.), Multiple-task performance. London: Taylor & Francis. 212 Kraemer, K.L., & Pinsonneault, A. (1990). Technology and groups: Assessment of the empirical research. In J. Galegher, R.E. Kraut, & C. Egido (Eds.), Intelligent teamwork. Hillsdale, NJ: Lawrence Erlbaum Associates. Kurimori, S., & Kakizaki, T. (1995). Evaluation of work stress using psychological and physiological measures of mental activity in a paced calculating task. Industrial Health, 33, 7-22. Kurstedt, H. (1993). Class notes ISE 4015. Virginia Polytechnic Institute and State University, Blacksburg, VA. Lane, J.D., Phillips-Bute, B.G., & Pieper, C.F. (1998). Caffeine raises blood pressure at work. Psychosomatic Medicine, 60(3), 327-330. Lanzetta, J.T. (1955). Group behavior under stress. Human Relations, 8, 29-52. Lanzetta, J.T., & Roby, T.B. (1960). The relationship between certain group process variables and group problem-solving efficiency. Journal of Social Psychology, 52, 135-148. Lanzetta, J.T., & Roby, T.B. (1965). Effect of work-group structure and certain task variables on group performance. Journal of Abnormal and Social Psychology, 53, 307-314. Laurell, H., & Lisper, H.O. (1976). Changes in subsidiary reaction time and heart-rate during car driving, passenger travel and stationary conditions. Ergonomics, 19(2), 149-156. Lazarus, R.S., & Folkman, S. (1984). Stress, appraisal, and coping. New York: Springer. Lewis, J.P. (2000). The project managers desk reference. New York: McGraw-Hill Liberatore, M.J., & Titus, G.J. (1983). The practice of management science in R&D project management. Management Science, 29(8), 962-974. Littell, R.C., Milliken, G.A., Stroup, W.W., and Wolfinger, R.D. (2002). SAS system for mixed models. Cary, NC: SAS Institute. Locke, E.A. (1968). Toward a theory of task motivation and incentives. Organizational Behavior and Human Performance, 3, 157-189. Locke, E.A. (1976). The nature and causes of job satisfaction. In M.D. Dunnette (Ed.), Handbook of industrial and organizational psychology. Chicago: Rand McNally. Lott, A.J., & Lott, B.E. (1965). Group cohesiveness as interpersonal attraction: a review of relationships with antecedent and consequent variables. Psychological Bulletin, 4, 259-309. Luczak, H., Beitz, W., Springer, J., & Langner, T. (1991). Frictions and frustrations in creative informatory work with computer aided design CAD-Systems. Advances in Human Factors/Ergonomics, 18A, 228-232. Malhotra, A., Thomas, J.C., Carroll, J.M., & Miller, L.A. (1980). Cognitive processes in design. International Journal of Man-Machine Studies, 12, 119-140. Marmel, E. (2000). The Microsoft Project 2000 bible. Foster City, CA: IDG Books Worldwide, Inc. Maslow, A.H. (1943). A theory of human motivation. Psychological Review, 50, 370-396. May, R. (1950). The meaning of anxiety. New York: Ronald Press. McGrath, J.E. (1970). A conceptual formulation for research on stress. In J.E. McGrath (Ed.), Social and psychological factors in stress. New York: Holt, Rinehart, and Winston. McGrath, J.E. (1984). Groups: interaction and performance. Englewood Cliffs, NJ: Prentice-Hall. 213 McGrath, J.E., & Hollingshead, A.B. (1994). Groups interacting with technology. Thousand Oaks: SAGE. McIntyre, R.M., Morgan, Jr., B.B., Salas, E., & Glickman, A.S. (1988). Teamwork from team training: new evidence for the development of teamwork skills during operational training. In Proceedings of the interservice/industry training equipment conference and exhibition. Alexandria, VA: American Defense Preparedness Association. McIntyre, R.M., & Salas, E. (1995). Measuring and managing for team performance: Lessons from complex environments. In R.A. Guzzo & E. Salas & Associates (Eds.), Team effectiveness and decision making in organizations. San Francisco: Jossey-Bass. McLeod, P.L. (1992). An assessment of the experimental literature on electronic support of group work: results of a meta-analysis. Human-Computer Interaction, 7, 281-314. McNair, D.G., Lorr, M., & Droppleman, L.F. (1992). Manual for the profile of mood states. San Diego, CA: Educational and Industrial Testing Service. McQuarrie, A.D., & Tsai, C. (1998). Regression and time series model selection. River Edge, NJ: World Scientific Publishing Co. Pte. Ltd. McRae, A.V. (1966). Interaction content and team effectiveness (Tech. Report 66-10). Alexandria, VA: The George Washington University Human Resources Research Office. Medsker, G.J., & Campion, M.A. (1997). Job and team design. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (pp. 450-489). New York: John Wiley & Sons. Meister, D. (1985). Behavioral analysis and measurement methods. New York: John Wiley & Sons. Meredith, J.W. (1997). Empirical investigation of sociotechnical issues in engineering design. Unpublished doctoral dissertation, Virginia Polytechnic Institute and State University, Blacksburg, VA. Meshkati, N. (1988). An eclectic and critical review of four primary mental workload assessment methods: a guide for developing a comprehensive model. In P.A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 251-267). North-Holland: Elsevier. Meshkati, N., & Loewenthal, A. (1988). The effects of individual differences in information processing behavior on experiencing mental workload and perceived task difficulty: A preliminary experimental investigation. In P.A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 269-284). North-Holland: Elsevier. Miller, F.B. (1958). Situational interactions a worthwhile concept? Human Organization, 17, 37-47. Moder, J.J., Phillips, C.R., & Davis, E.W. (1983). Project management with CPM, PERT, and procedure diagramming. New York: Van Nostrand Reinhold Company. Moore, H.G. (1962). The effects of load and accessibility of information upon performance of small groups. Unpublished doctoral dissertation, University of Michigan, Ann Arbor, MI. Moray, N. (1988). Mental workload since 1979. International Review of Ergonomics, 2, 123-150. Morgan, Jr., B.B., & Bowers, C. (1995). Teamwork stress: Implications for team decision making. In R.A. Guzzo and E. Salas (Eds.) Team effectiveness and decision making organizations. San Francisco, CA: Jossey Bass. Morgan, Jr., B.B., Glickman, A.S., Woodard, E.A., Blaiwes, A. & Salas, E. (1986). Measurement of team behaviors in a navy environment (NTSC-TR-86-014). Orlando, FL: Naval Training Systems Center. 214 Nagamachi, M., & Imada, A.S. (1992). A macroergonomic approach for improving safety and work design. In Proceedings of the human factors society 36th annual meeting (pp. 859-861). Santa Monica, CA: Human Factors Society. Nair, S.N., Hoag, D.W., Leonard, D.C., Sharit, J., & Czaja, S.J. (1997). Effects of age and experience on mental workload for computer tasks. In Proceedings of the human factors and ergonomics society 41st annual meeting (pp.139-143). Santa Monica, CA: Human Factors and Ergonomics Society. Naylor, J.C., & Briggs, G.E. (1965). Team training effects under various conditions. Journal of Applied Psychology, 49, 223-229. Neck, C., Manz, C., & Anad, V. (2000). Self managing teams in a crystal ball. In M.M. Beyerlein (Ed.), Work teams: Past, present, and future. Dordrecht: Kluwer Academic Press. Neter J., Kutner, M.H., Nachtsheim, C.J., & Wasserman, W. (1996). Applied linear statistical models. Chicago: Irwin. Newell, A., & Simon, H.A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice Hall. Nieva, V.F., Fleishman, E.A., & Rieck, A. (1978, 1985 reprint). Team dimensions: Their identity, their measurement and their relationships. Bethesda, MD: Advanced Research Resources Organization. Norman, D.A. (1988). Design of everyday things. New York: Doubleday/Currency. Northhouse, P.G. (2004). Leadership theory and practice, 3rd Ed. Thousand Oaks, CA: SAGE. Nygren, T.E. (1991). Psychometric properties of subjective workload measurement techniques: Implications for their use in the assessment of perceived mental workload. Human Factors, 33(1), 17-33. OBrien, G. (1968). The measurement of cooperation. Organizational Behavior and Human Performance, 3, 427-439. OConaill, B., Whittaker, S., & Wilber, S. (1993). Conversations over video conferences: an evaluation of the spoken aspects of video-mediated communication. Human-Computer Interaction, 8, 389-428. OHanlon, J.F., & Kelley, G.R. (1977). Comparison of performance and physiological changes between drivers who perform well and poorly during prolonged vehicular operation. In R.R. Mackie (Ed.), Vigilance: Theory, operational performance, and physiological correlates. New York: Plenum Press. Ochsman, R.B., & Chapanis, A. (1974). The effects of 10 communication modes on the behavior of teams during co-operative problem solving. International Journal of Man-Machine Studies, 6(5), 579619. O'Donnell, R.D., & Eggemeier, F.T. (1986). Workload assessment methodology. In K.R. Boff, L. Kaufman, & J.P. Thomas (Eds.), Handbook of perception and human performance Vol. 2. New York: John Wiley and Sons. Olarian, B.A. (1995). Perceived communication outcomes in computer-mediated communication: An analysis of three systems among new users. Information Processing & Management, 31(4), 525541. Olson, G.M., Olson, J.S., Carter, M.R., & Storrsten, M. (1992). Small group design meetings: An analysis of collaboration. Human-Computer Interaction, 7, 347-374. Orasanu, J., & Salas, E. (1993). Team decision making in complex environments. In G.A. Klein, J. Orasanu, R. Calderwood, & C.E. Zsambok (Eds.), Decision making in action: Models and methods (pp. 327-345). Norwood, NJ: Ablex. 215 O'Reilly, C.A., & Roberts, K.H. (1977). Task group structure, communication, and effectiveness in three organizations. Journal of Applied Psychology, 62(6), 674-681. Oser, R., McCallum, G.A., Salas, E., & Morgan, Jr., B.B. (1989). Toward a definition of teamwork: An analysis of critical team behaviors (NTSC 89-004). Orlando, FL: Naval Training Systems Center. Paas, F.G.W.C., & Adam, J.J. (1991). Human information processing during physical exercise. Ergonomics, 34(11), 1385-1397. Pahl, G., & Beitz, W. (1996). Engineering design a systematic approach, London: Springer. Pasmore, W. (1988). Designing effective organizations: the sociotechnical systems perspective. New York: Wiley & Sons. Pasmore, W., Francis, C., Haldeman, J., & Shani, A. (1982). Sociotechnical systems: A North American reflection on empirical studies of the seventies. Human Relations, 35(12), 1179-1204. Pearce, C.L., & Conger, J.A. (2003). Shared leadership: Reframing the hows and whys of leadership. Thousand Oaks, CA: Sage Publications. Perrow, C. (1967). A framework for the comparative analysis of organizations. American Sociological Review, April, 194-208. Petty, M.M., McGee, G.W., & Cavender, J.W. (1984). A meta-analysis of the relationships between individual job satisfaction and individual performance. Academy of Management Review, 9(4), 712-721. Pinsonneault, A., & Kraemer, K.L. (1990). The effects of electronic meetings on group processes and outcomes: An assessment of the empirical research. European Journal of Operational Research, 42(2), 143-161. Pinto, M.B., & Pinto, J.K. (1991). Determinants of cross-functional cooperation in the project implementation process. Project Management Journal, 12(2), 13-19. Pinto, M.B., Pinto, J.K., & Prescott, J.E. (1993). Antecedents and consequences of project team crossfunctional cooperation. Management Science, 39(10), 1281-1297. Pollack-Johnson, B., & Liberatore, M.J. (1998). Project management software usage patterns and suggested research directions for future developments. Project Management Journal, 29(2), 1928. Porges, S.W., & Byrne, E.A. (1992). Research methods for measurement of heart rate and respiration. Biological Psychology, 34, 93-130. Price, H.E. (1985). The allocation of functions in systems. Human Factors, 27(1), 33-45. Prince, C., Chidester, T.R., Bowers, C., & Cannon-Bowers, J. (1992). Aircrew coordination achieving teamwork in the cockpit. In R.W. Swezey & E. Salas (Eds.) Teams: Their training and performance. Norwood, NJ: ABLEX. Quinn, R.P., & Sheppard, L.J. (1974). The 1972-73 quality of employment survey. Ann Arbor, MI: University of Michigan. Ragusa, J.M., & Bochenek, G. (1996). Product design using virtual environments a conceptual evolutionary approach. In International Engineering Management Conference, (pp. 494-499) Piscataway, NJ: IEEE. Reddy, Y.V.R., Srinivas, K., Jagannathan, V., & Karinthi, R. (1993). Computer support for concurrent engineering. Computer, 26(1), 12-16. 216 Robbins, S.R. (1983). Organizational theory: the structure and design of organizations. Englewood Cliffs, NJ: Prentice-Hall. Roby, T.B., & Lanzetta, J.T. (1957). Conflicting principles in man-machine system design. Journal of Applied Psychology, 41(3), 170-178. Rodahl, K. (1989). The physiology of work. London: Taylor and Francis. Rohmert, W. (1977). Human work taxonomy for system applications. In N. Moray (Ed.), Mental workload: Its theory and measurement. New York: Plenum Press. Rokeach, M. (1973). The nature of human values. New York: Free Press. Rolfe, J.M., & Lindsay, S.J.E. (1973). Flight deck environment and pilot workload: biological measures of workload. Applied Ergonomics, 4(4), 199-206. Rooney, E.F., Morency, R.R., & Herrick, D.R. (1993). Macroergonomics and total quality management at L.L. Bean: A case study. In R. Neilsen, & K. Jorgensen (Eds.), Advances in industrial ergonomics and safety (pp.493-498). London: Taylor & Francis. Roscoe, A.H. (1993). Heart rate as a psychophysiological measure for in-flight workload assessment. Ergonomics, 36(9), 1055-1062. Rotter, G.S., & Portugal, S.M. (1969). Group and individual effects in problem solving. Journal of Applied Psychology, 53(4), 338-341. Rouse, W.B., & Cody, W.J. (1986). Function allocation in manned system design. In Proceedings of the IEEE international conference on systems, man, and cybernetics. Piscataway, NJ: IEEE. Saito, K. (1999). Measurement of fatigue in industries. Industrial Health, 37, 134-142. Salas, E., & Cannon-Bowers, J. (2000). Teams in organizations. In M.M. Beyerlein (Ed.), Work teams: Past, present, and future. Dordrecht: Kluwer Academic Publishers. Salas, E., Dickinson, T.L., Converse, S.A., & Tannenbaum, S.I. (1992). Toward an understanding of team performance and training. In R.W. Swezey & E. Salas (Eds.), Teams: Their training and performance. Norwood, NJ: ABLEX. Salas, E., Driskell, J.E., & Hughes, S. (1996). Introduction: The study of stress and human performance. In J.E. Driskell & E. Salas (Eds.), Stress and human performance. Mahwah, NJ: Lawrence Erlbaum. Salas, E., Morgan, B.B., & Glickman, A.S. (1987). The evolution and maturation of operational teams in a training environment. In Proceedings of the human factors society 31st annual meeting (pp. 8286). Santa Monica, CA: Human Factors Society. Salvendy, G. (1981). Classification and characteristics of paced work. In Machine pacing and occupational stress (pp. 5-12). London: Taylor and Francis. Sanders, M.S., & McCormick, E.J. (1993). Human factors in engineering and design. New York: McGraw-Hill. Schachter, S., Ellertson, N., McBride, D., & Gregory, D. (1951). An experimental study of cohesiveness and productivity. Human Relations, 4, 229-238. Schon, D.A. (1992). Designing as reflective conversation with the materials of a design situation. Research in Engineering Design, 3, 131-141. Schwartz, J.E., Jandorf, L., & Krupp, L.B. (1993). The measurement of fatigue: A new instrument. Journal of Psychosomatic Research, 37(7), 753-762. 217 Seashore, S.E. (1977). Group cohesiveness in the industrial work group. New York: ARNO Press. Selye, H. (1956). The stress of life. New York: McGraw-Hill. Serfaty, D., & Kleinman, D.L. (1990). Adaption processes in team decision making and coordination. In Proceedings of the IEEE international conference on systems, man and cybernetics (pp. 394395). Piscataway, NJ: IEEE. Serfaty, D., Entin, E.E., & Volpe, C. (1993). Adaptation to stress in team decision-making and coordination. In Proceedings of the human factors and ergonomics society (pp. 1228-1232). Santa Monica, CA: Human Factors and Ergonomics Society. Shacham, S. (1983). A shortened version of the profile of mood states. Journal of Personality Assessment, 47, 305-306). Sharit, J., & Salvendy, G. (1982). Occupational stress: review and reappraisal. Human Factors, 24(2), 129-162. Sharit, J., Czaja, S.J., Nair, S.N., Hoag, D.W., Leonard, D.C., & Dilsen, E.K. (1998). Subjective experiences of stress, workload, and bodily discomfort as a function of age and type of computer work. Work and Stress, 12(2), 125-144. Shaw, M.E. (1976). Group dynamics: The psychology of small group behavior, New York: McGraw-Hill. Sheridan, T.B. (1979). Mental workload in decision and control. In Proceedings of the IEEE conference on decision and control (pp. 977-982). New York: IEEE. Shiflett, S., Eisner, E.J., Price, S.J., & Schemmer, F.M. (1985). The definition and measurement of small military unit team functions (Report 1397). Fort Benning, GA: Research Institute for the Behavioral Sciences. Short, J., Williams, E., & Christie, B. (1976). The social psychology of telecommunications. New York: McGraw-Hill. Shure, G.H., Rogers, M.S., Larsen, I.M., & Tassone, J. (1962). Group planning and task effectiveness. Sociometry, 25(3), 263-282. Siegel, A.I., & Federman, P.J. (1973). Communications content training as an ingredient in effective team performance. Ergonomics, 16, 403-416. Siegel, J., Dubrovsky, V., Kiesler, S., & McGuire, T.W. (1986). Group processes in computer-mediated communication. Organizational Behavior and Human Decision Processes, 37, 157-187. Simon, H.A. (1973). The structure of ill structured problems. Artificial Intelligence, 4, 181-201. Sirevaag, E.J., Kramer, A.F., Wickens, C.D., Reisweber, M., Strayer, D.L., & Grenell, J.F. (1993). Assessment of pilot performance and mental workload in rotary wing aircraft. Ergonomics, 36(9), 1121-1140. Smith, D.L., Petruzello, S.J., Kramer, J.M., Warner, S.E., Bone, B.G., & Misner, J.E. (1995). Selected physiological and psychobiological responses to physical activity in different configurations of firefighting gear. Ergonomics, 38(10), 2065-2077. Smith, H.W. (1989). Group versus individual problem solving and type of problem solved. Small Group Research, 20(3), 357-366. Smith, J., & Vanecek, M.T. (1989). A nonsimultaneous computer conference as a component of group decision support systems. In 22nd annual Hawaii international conference on system sciences (pp. 370-377). Piscataway, NJ: IEEE. 218 Smith, P.C., & Kendall, L.M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149-155. Spector, P.E. (1997). Job satisfaction: Application, assessment, cause, and consequences. Thousand Oaks: SAGE. Spector, P.E., Dwyer, D.J., & Jex, S.M. (1988). Relation of job stressors to affective health, and performance outcomes: A comparison of multiple data sources. Journal of Applied Psychology, 73(1), 11-19. Springer, J., Langner, T., Luczak, H., & Beitz, W. (1991). Experimental comparison of CAD systems by stressor variables. International Journal of Human Computer Interaction, 3(4), 375-405. Sproull, L., & Kiesler, S. (1986). Reducing social context cues: electronic mail in organizational communication. Management Science, 32(11), 1492-1512. Steiner, I.D. (1972). Group process and productivity, New York: Academic Press. Stern, J.A., Boyer, D., & Schroeder, D. (1994). Blink rate: A possible measure of fatigue. Human Factors, 36(2), 285-297. Stogdill, R.M. (1972). Group productivity, drive, and cohesiveness. Organizational Behavior and Human Performance, 8, 26-43. Stout, R.J., Salas, E., & Carson, R. (1994). Individual task proficiency and team process behavior: Whats important for team functioning? Military Psychology, 6(3), 177-192. Straus, S.G. (1997). Technology, group process, and group outcomes: testing the connections in computer-mediated and face-to-face groups. Human-Computer Interaction, 12, 227-266. Svensson, E., Angelborg-Thanderz, M., & Sjoberg, L. (1993). Mission challenge, mental workload and performance in military aviation. Aviation Space and Environmental Medicine, 64(11), 985-991. Sundstrom, E., De Meuse, K.P., & Futrell, D. (1990). Work teams: Applications and effectiveness. American Psychologist, 45(2), 120-133. Swezey, R.W., & Salas, E. (1992). Guidelines for use in team-training development. In R.W. Swezey & E. Salas (Eds.), Teams: Their training and performance. Norwood, NJ: ABLEX. Tang, J.C. (1991). Findings from observational studies of collaborative work. International Journal of Man-Machine Studies, 34, 143-160. Tattersall, A.J., & Foord, P.S. (1996). An experimental evaluation of instantaneous self-assessment as a measure of workload. Ergonomics, 39(5), 740-748. Taylor, A.J. (1993). The parallel nature of design. Journal of Engineering Design, 4(2), 141-152. Taylor, J.C., & Felten, D.F. (1993). Performance by design: sociotechnical systems in North America, Englewood Cliffs, NJ: Prentice Hall. Temple, J.G., Warm, J.S., Dember, W.N., Jones, K.S., LaGrange, C.M., & Matthews, G. (2000). The effects of signal salience and caffeine on performance, workload, and stress in an abbreviated vigilance task. Human Factors, 42(2), 183-194. Thackray, R.I., Bailey, J.P., & Touchstone, R.M. (1977). Physiological, subjective, and performance correlates of reported boredom and monotony while performing a simulated radar control task. In. R.R. Mackie (Ed.), Vigilance: theory, operational performance, and physiological correlates (p. 203-216). New York: London. Tjosvold, D., & Johnson, D. (2000). Deutschs theory of cooperation and competition. In M.M. Beyerlein (Ed.), Work teams: Past, present and future. Dordrecht: Kluwer Academic Publishers. 219 Torre, Jr., J.P., Wanasack, S., Hudgens, G.A., King, J.M., Fatkin, L.T., Mazurczak, J., & Breitenbach, J. (1991). Effects of competition and mode of fire on physiological responses, psychological stress reactions, and shooting performance. Technical Memorandum 11-91. Aberdeen Proving Ground, MD: Human Engineering Laboratory. Townsend, A.M., DeMarie, S.M., & Hendrickson, A.R. (2000). Virtual teams: technology and the workplace of the future. IEEE Engineering Management Review, 28(2), 69-80. Trist, E.L. (1978). In W.A. Pasmore & J.J. Sherwood (Eds.), Sociotechnical systems. La Jolla, CA: University Assoc. Trist, E.L., and Bamforth, K.W. (1951). Some social and psychological consequences of the longwall method of coal-getting. Human Relations, 4, 3-38. Trumbull, R., & Appley, M.H. (1986). A conceptual model for the examination of stress dynamics. In M.H. Appley and R. Trumbull (Eds.), Dynamics of stress. New York: Plenum Press. Tsang, P., & Wilson, G.F. (1997). Mental workload. In G. Salvendy (Ed.), The handbook of human factors and ergonomics (pp. 417-449). New York: Wiley. Tuckman, B.W. (1965). Developmental sequences in small groups. Psychological Bulletin, 63, 384-399. Tuckman, B.W., & Jensen, M. (1977). Stages of small group development revisited. Group and Organizational Studies, 2, 419-427. Tuckman, J., & Lorge, I. (1965). Group processes, structure, and productivity. In I.D. Steiner & M. Fishbein (Eds.), Current studies in social psychology (pp. 409-416). New York: Holt, Rinehart, and Winston. Ullman, D.G, Wood, S., & Craig, D. (1990). The importance of drawing in the mechanical design process. Computers and Graphics, 14(2), 263-274. Ullman, D.G., Stauffer, L.A., & Dietterich, T.G. (1987). Toward expert CAD. Computers in Mechanical Engineering, 6(3), 56-65, 70. Urban, J.M., Bowers, C.A., Monday, S.D., & Morgan, B.B. (1993). Effects of workload on communication processes in decision making teams. In Proceedings of the human factors and ergonomics society 37th annual meeting (pp. 1233-1237). Santa Monica, CA: Human Factors and Ergonomics Society. Urban, J.M., Bowers, C.A., Monday, S.D., & Morgan, B.B. (1995). Workload, team structure, and communication in team performance. Military Psychology, 7(2), 123-139. Urban, J.M., Bowers, C.A., Morgan, B.B., & Maniam, N. (1994). The effects of workload and uncertainty on team development. In Proceedings of the human factors and ergonomics society 38th annual meeting (pp. 1234-1237). Santa Monica, CA: Human Factors and Ergonomics Society. Urban, J.M., Bowers, C.A., Morgan, B.B., Braun, C.C., & Kline, P.B. (1992). The effects of hierarchical structure and workload on the performance of team and individual tasks. In Proceedings of the human factors and ergonomics society 36th annual meeting (pp. 829-833). Santa Monica, CA: Human Factors and Ergonomics Society. Urban, J.M., Weaver, J.L., Bowers, C.A., & Rhodenizer, L. (1996). Effects of workload and structure on team processes and performance: Implications for complex team decision making. Human Factors, 38(2), 300-310. Veltman, J.A., & Gaillard, W.K. (1993). Indices of mental workload in a complex task environment. Neuropsychobiology, 28(1-2), 72-75. 220 Vicente, K.J., Thornton, D.C., & Moray, N. (1987). Spectral analysis of sinus arrhythmia: A measure of mental effort. Human Factors, 29(2), 171-182. Volow, M.R., & Erwin, C.W. (1973). The heart rate variability correlates of spontaneous drowsiness onset. In International automotive engineering congress. New York: ASME. Vroom, V.H. (1982). Work and motivation. Malabar, Fl: Robert E. Krieger. Walther, JB. (1992). Interpersonal effects in computer-mediated interaction. Communication Research, 19(1), 52-90. Walton, R.E., & Hackman, J.R. (1986). Groups under contrasting management strategies. In P.S. Goodman (Ed.), Designing effective work groups. San Francisco: Joey-Bass. Wastell, D. (1991). Physiological measurement of cognitive load during interaction with process control displays. Advances in Human Factors/Ergonomics, 18A, 228-232. Weeks, G.D., & Chapanis, A. (1976). Cooperative versus conflictive problem solving in three telecommunication modes. Perceptual and Motor Skills, 42, 879-917. Weiss, J.W., & Wysocki, R.K. (1992). 5-phase project management: a practical planning and implementation guide. Reading, MA: Addison Wesley. Welch, D.L. (1994). Design enhancement through human factored function allocation. In IEEE/AIAA 13th annual digital avionics systems conference (pp. 318-321), Piscataway, NJ: IEEE. Whitaker, L.A., Hohne, J., & Birkmire-Peters, D.P. (1997). Assessing cognitive workload metrics for evaluating telecommunication tasks. In Proceedings of the human factors and ergonomics society 41st annual meeting (pp. 325-329). Santa Monica, CA: Human Factors and Ergonomics Society. Wickens, C.D. (1979). Measures of workload, stress, and secondary tasks. In N. Moray (Ed.), Mental workload: Its theory and measurement. New York: Plenum Press. Wickens, C.D. (1984). Engineering psychology and human performance. Glenview, IL: Scott, Foresman and Company. Wierwille, W.W., & Casali, J.G. (1983). A validated rating scale for global mental workload measurement applications. In Proceedings of the human factors society 27th annual meeting (pp. 129-133). Santa Monica, CA: Human Factors Society. Wierwille, W.W., & Connor, S.A. (1983). Evaluation of 20 workload measures using a psychomotor task in a moving-base aircraft simulator. Human Factors, 25(1), 1-16. Wierwille, W.W., & Eggemeier, F.T. (1993). Recommendations for mental workload measurement in a test and evaluation environment. Human Factors, 35(2), 263-281. Wierwille, W.W., & Moto, W.H. (1981). Significant changes in driver-vehicle response measure for extended duration simulation driving tasks. In Proceedings of the first European annual conference on human decision making and manual control (pp. 298-314). Delft, Netherlands: Delft University of Technology. Wierwille, W.W., Rahimi, M., & Casali, J.G. (1985). Evaluation of 16 measures of mental workload using a simulated flight task emphasizing mediational activity. Human Factors, 27(5), 489-502. Wilke, H. A.M., & Meertens, R.W. (1994). Group performance. London: Routledge. Williges, R.C., & Wierwille, W.W. (1979). Behavioral measures of aircrew mental workload. Human Factors, 21(5), 549-574. Williges, R.C., Johnston, W.A., & Briggs, G.E. (1966). Role of verbal communication in teamwork. Journal of Applied Psychology, 50(6), 473-478. 221 Wilson, G.F., Fullenkamp, P., & Davis, I. (1994). Evoked potential, cardiac, blink, and respiration measures of pilot workload in air-to-ground missions. Aviation Space and Environmental Medicine, 65(2), 100-105. Wilson, P. (1991). Computer supported cooperative work (CSCW): origins, concepts and research initiatives. Computer Networks and ISDN Systems, 23, 91-95. Yaloms, I.D. (1985). The theory and practice of group psychotherapy, 3rd ed. New York: Basic Books. Zaccaro, S.J., & Lowe, C.A. (1988). Cohesiveness and performance on an additive task: evidence for multidimensionality. Journal of Social Psychology, 128(4), 547-558. Zajonc, R.B. (1965). Social facilitation. Science, 149, 269-274. Zajonc, R.B. (1968). Social facilitation. In D. Cartwright & A. Zander (Eds.), Group dynamics: Research and theory. New York: Harper & Row. 222 . Appendices 223
MOST POPULAR MATERIALS FROM LIB
MOST POPULAR MATERIALS FROM Virginia Tech