by Copyright Lilin Liang 2005
The Dissertation Committee for Lilin Liang Certifies that this is the approved version of the following dissertation:
Small Project Benchmarking
Committee: G. Edward Gibson, Jr., Supervisor Stephen R. Thomas, Co-Supervisor James T. OConnor John D. Borcherding William R. Kelly
Small Project Benchmarking
by Lilin Liang, B.S., M.S.
Dissertation
Presented to the Faculty of the Graduate School of The University of Texas at Austin in Partial Fulfillment of the Requirements for the Degree of
Doctor of Philosophy
The University of Texas at Austin December 2005
Dedication
To my dearest family, for their unconditional love and support with the greatest appreciation.
Acknowledgements
I am forever indebted to my supervisor Dr. G. Edward Gibson, Jr. and Dr. Stephen R. Thomas, for their kind support and guidance. Without their insight and supervision, this research work could not have been completed. I would like to extend my thanks to Dr. G. Edward Gibson, Jr., who provided invaluable support, and guidance in my completion of the Dispute Resolution Portfolio Program (DRPP) under the Center for Public Policy Dispute Resolution at the Law school. Furthermore, I would like to also acknowledge Dr. Stephen R.
Thomas for providing me comments and encouragements with great enthusiasm. I would like to acknowledge the support and feedback from the Small Project Research Team, who provided the industry background necessary for this research. I would also like to extend my appreciation to Deborah L. DeGezelle
and Frances V. Decoux for their invaluable assistance and support. I am also grateful to Tina, for her companion and encouragements. Finally, I owe my deepest gratitude to my parents, for their unconditional love and support during my graduate studies.
v
Small Project Benchmarking
Publication No._____________
Lilin Liang, Ph.D. The University of Texas at Austin, 2005
Supervisors:
G. Edward Gibson, Jr., Stephen R. Thomas
Benchmarking is recognized as an important function in improving practices and therefore capital effectiveness. In 1999, the Construction Industry
Institute (CII) defined benchmarking as a systematic process of measuring ones performance against recognized leaders for the purpose of determining best practices that lead to superior performance when adopted and utilized. It is conservatively estimated that 40 to 50 percent of todays industry capital budgets are spent on small projects, making successful small project execution especially important. However, many companies deliver small
projects with comparatively reduced capital effectiveness. The primary reasons are the combination of compressed project life cycle and less management support due to low visibility. Research investigating small project execution has been insufficient up to this point; however, it is evident that providing general
vi
practices for small project execution and the validation of their value will contribute to the overall effectiveness of the construction industry. Data from 194 small capital projects were collected and analyzed for this research. This research (1) identifies the differences between large and small projects; (2) utilizes newly collected information to provide industry norms for small project metrics; (3) evaluates the impact of practice use for small project execution; and, lastly, (4) develops a sustainable system to collect and continuously benchmark small project data.
vii
Table of Contents
List of Tables........................................................................................................xiii List of Figures ....................................................................................................... xv Chapter 1 Introduction ............................................................................................ 1 1.1. The Construction Industry Institute....................................................... 2 1.1.1. Benchmarking & Metrics Program ......................................... 3 1.2. Problem Statement ................................................................................ 6 1.3. Research Objectives .............................................................................. 7 1.4. Research Hypothesis ............................................................................. 8 1.5. Research Scope ................................................................................... 10 1.6. Organization of Dissertation ............................................................... 11 Chapter 2 Background........................................................................................... 12 2.1. Benchmarking in Construction Industry ............................................. 12 2.2. Small Project Definition...................................................................... 13 2.3. Small Project Management ................................................................. 16 2.4. Project Success Factors ....................................................................... 17 2.5. Project Performance and Practice Use Correlations............................ 19 2.6. Literature Review Summary ............................................................... 21 Chapter 3 Research Methodology ......................................................................... 23 3.1. Questionnaire Development ................................................................ 24 3.2. Questionnaire Pilot Testing................................................................. 25 3.3. Data Collection and Validation ........................................................... 26 3.4. Data Analysis Techniques ................................................................... 27 3.4.1. 3.4.2. 3.4.3. Box and Whisker Diagrams .................................................. 28 Linear Regression Analysis................................................... 30 One-Way Analysis of Variance............................................. 33 viii
3.5. Limitation and Biases.......................................................................... 34 Chapter 4 The Small Project Questionnaire Measuring Project Performance, Practice Use and Other Factors .................................................................... 36 4.1. General Information Section ............................................................... 36 4.2. Project Performance Section ............................................................... 38 4.2.1. 4.2.2. 4.2.3. 4.2.4. 4.3.1. 4.3.2. 4.3.3. 4.3.4. 4.3.5. 4.3.6. 4.3.7. 4.3.8. 4.3.9. Project Cost Performance...................................................... 38 Project Schedule Performance .............................................. 39 Project Change Performance ................................................. 40 Project Safety Performance ................................................... 41 Front-End Planning Practice ................................................. 43 Design Practice...................................................................... 46 Procurement Practice............................................................. 49 Construction Practice ............................................................ 52 Start-up Practice Use ............................................................. 55 Organization Practice ............................................................ 56 Project Processes Practice ..................................................... 58 Project Controls Practice ....................................................... 60 Safety, Health and Environment Practice Use ...................... 62
4.3. Practice Use Questions........................................................................ 42
4.3.10. Automation/Integration Practice ........................................... 63 Chapter 5 Descriptive Data ................................................................................... 64 5.1. Descriptive Statistics ........................................................................... 64 5.1.1. 5.1.2. 5.1.3. 5.1.4. 5.2.1. Respondent Type................................................................... 68 Project Location .................................................................... 68 Industry Group ...................................................................... 70 Project Nature........................................................................ 72 Cost Performance Metrics ..................................................... 73 ix
5.2. Project Performance ............................................................................ 73
5.2.2. 5.2.3. 5.2.4. 5.2.5.
Schedule Performance Metrics.............................................. 79 Project Change Performance Metrics.................................... 82 Safety Performance Metrics .................................................. 84 Phase Factors......................................................................... 89
Chapter 6 Practice Use Results ............................................................................. 98 6.1. Practice Use Index Development ........................................................ 98 6.1.1. 6.1.2. 6.2.1. 6.2.2. 6.2.3. 6.2.4. 6.2.5. 6.2.6. 6.2.7. 6.2.8. 6.2.9. Reliability ............................................................................ 102 Validity................................................................................ 104 Front-End Planning ............................................................. 105 Design.................................................................................. 107 Procurement ........................................................................ 109 Construction ........................................................................ 110 Start-up & Commissioning.................................................. 112 Organization ........................................................................ 113 Project Processes ................................................................. 115 Project Controls................................................................... 117 Safety, Health & Environment ............................................ 119
6.2. Practice Use Index Data Presentation ............................................... 105
6.2.10. Automation/Integration Technology ................................... 120 Chapter 7 Data Relationships.............................................................................. 123 7.1. Difference Between Small And Large Projects ................................ 123 7.1.1. 7.1.2. 7.1.3. Selected Owner Comparisons ............................................. 128 Selected Contractor Comparisons ....................................... 132 Section Summary ................................................................ 137
7.2. Bi-variate Regression Analysis of Project Cost Growth and Practice Use Index............................................................................. 143 7.2.1. 7.2.2. Data Preparation and Model Diagnostics............................ 143 Owner Regression Results .................................................. 147 x
7.2.3. 7.3.1. 7.3.2.
Contractor Regression Results ............................................ 151 Owner Test Results ............................................................. 156 Contractor Test Results ....................................................... 159
7.3. T-test Comparison of Project Cost Growth and Practice Use Index. 155
7.4. Bi-variate Regression Analysis of Project Performance and Combined Practice Use Index ........................................................... 161 7.4.1. 7.4.2. 7.4.3. 7.4.4. Correlation among Practice Use Indices ............................. 162 Development of the Combined Indices ............................... 165 Owner Regression Results .................................................. 170 Contractor Regression Results ............................................ 172
7.5. T-test Comparison of Project Performance and Combined Practice Use Index........................................................................................... 174 7.5.1. 7.5.2. Owner Test Results ............................................................. 174 Contractor Test Results ....................................................... 176
7.6. Summary ........................................................................................... 177 Chapter 8 Conclusions and Recommendations ................................................... 179 8.1. Review of Research Objectives......................................................... 179 8.1.1. 8.1.2. 8.1.3. 8.1.4. Review existing research on small project management..... 180 Provide a consensus definition of a small project for use in this research......................................................................... 180 Summarize the differences in project performance and characteristics of large and small capital projects............... 181 Provide project norms, i.e., statistical summaries, for project performance and practice use metrics from the data collected for this research............................................ 182 Identify key factors, i.e., project management practices, required for successful small project execution. ................. 182 Provide guidance to project practitioners for improved small project management. ................................................. 182
8.1.5. 8.1.6.
8.2. Review of Research Hypothesis and Conclusions ............................ 184 xi
8.3. Research Contributions ..................................................................... 185 8.4. Future Research Opportunities and Recommendations .................... 187 Appendix A Appendix B Appendix C Appendix D Appendix E Appendix F Appendix G The Small Project Questionnaire ................................................. 189 Small Project Definition Analysis ............................................... 217 Company Bias Analysis............................................................... 226 Statistical Outputs for Coefficient Alpha .................................... 236 T-tests for Large and Small Projects Comparisons...................... 257 Regression Diagnostic Plots......................................................... 262 Compositional Difference Diagnostics........................................ 277
Bibliography........................................................................................................ 286 Vita .................................................................................................................... 291
xii
List of Tables
Table 1: Table 2: Table 3: Composition of Small Project Team .................................................. 5 Small Project Database by Participating Organization .................... 66 Central Tendency of Small Project Work Hours by Respondent Type.................................................................................................. 87 Table 4: Table 5: Table 6: Table 7: Table 8: Table 9: Table 10: Table 11: Table 12: Table 13: Key Statistics for Owner Phase Cost Factors................................... 91 Key Statistics for Contractor Phase Cost Factors............................. 92 Key Statistics for Owner Phase Duration Factors ............................ 95 Key Statistics for Contractor Phase Duration Factors...................... 96 Standardized Coefficient Alpha Summary by Respondent Type... 103 Front-End Planning Practice Use Statistics by Respondent Type.. 107 Design Practice Use Statistics by Respondent Type ...................... 108 Procurement Practice Use Statistics by Respondent Type............. 110 Construction Practice Use Statistics by Respondent Type............. 111 Start-up & Commissioning Practice Use Statistics by Respondent Type................................................................................................ 113 Table 14: Table 15: Table 16: Table 17: Organization Practice Use Statistics by Respondent Type ............ 115 Project Processes Practice Use Statistics by Respondent Type ..... 117 Project Controls Practice Use Statistics by Respondent Type ....... 118 Safety, Health & Environment Practice Use Statistics by Respondent Type............................................................................ 120 Table 18: Automation / Integration Technology Practice Use Statistics by Respondent Type............................................................................ 122 xiii
Table 19: Table 20: Table 21: Table 22: Table 23: Table 24: Table 25: Table 26: Table 27: Table 28: Table 29: Table 30: Table 31: Table 32: Table 33: Table 34: Table 35: Table 36:
Characteristics of the Small Project Dataset .................................. 124 Characteristics of the Large Project Dataset .................................. 125 Summary of Differences in Owner Performance Variation........... 138 Summary of Differences in Contractor Performance Variation..... 138 Summary of Differences in Owner Practice Use Metrics .............. 140 Summary of Differences in Contractor Practice Use Metrics........ 141 Owner Projects Practice Use Indices Regression Summary .......... 148 Contractor Projects Practice Use Indices Regression Summary .... 151 Owner Projects Practice Use Indices T-test Summary................... 157 Contractor Projects Practice Use Indices T-test Summary ............ 159 Owner Practice Use Index Correlation Matrix............................... 162 Contractor Practice Use Index Correlation Matrix......................... 163 Owner Exploratory Factor Analysis Loading Matrix .................... 167 Composition of Practice Use Indices ............................................. 169 Owner Combined Practice Use Indices Regression Summary ...... 171 Contractor Combined Practice Use Indices Regression Summary 173 Owner Combined Practice Use Indices T-test Summary............... 175 Contractor Combined Practice Use Indices T-test Summary......... 176
xiv
List of Figures
Figure 1: Figure 2: Figure 3: Figure 4: Figure 5: Figure 6: Figure 7: Figure 8: Figure 9: Figure 10: Figure 11: Figure 12: Figure 13: Figure 14: Figure 15: Figure 16: Figure 17: Figure 18: Figure 19: Figure 20: Figure 21: Research Methodology..................................................................... 23 Sample Box and Whisker Diagram.................................................. 29 BM&M database by Project Size and Questionnaire Used.............. 65 Small Project Database by Respondent Type .................................. 68 Small Project Database by Project Location .................................... 69 Owner versus Contractor Participation by Project Location............ 69 Small Project Database by Industry Group ...................................... 70 Small Project Database by Project Nature ....................................... 72 Distribution of Project Cost Growth by Respondent Type .............. 75 Distribution of Project Budget Factor by Respondent Type ............ 78 Distribution of Project Schedule Growth by Respondent Type ....... 80 Distribution of Project Schedule Factor by Respondent Type ......... 82 Distribution of Change Cost Factor by Respondent Type ............... 83 Data Distribution of TRIR by Respondent Type ............................. 86 Data Distribution of DART Rate by Respondent Type ................... 88 Phase Cost Factors by Respondent Type ......................................... 90 Phase Duration Factors by Respondent Type .................................. 94 Sample Calculation for Practice Use Index.................................... 100 Front-End Planning Practice Use Index by Respondent Type ....... 106 Design Practice Use Index by Respondent Type ........................... 108 Procurement Practice Use Index by Respondent Type .................. 109 xv
Figure 22: Figure 23:
Construction Practice Use Index by Respondent Type .................. 111 Start-up & Commissioning Practice Use Index by Respondent Type................................................................................................ 112
Figure 24: Figure 25: Figure 26: Figure 27:
Organization Practice Use Index by Respondent Type.................. 114 Project Processes Practice Use Index by Respondent Type........... 116 Project Controls Practice Use Index by Respondent Type............. 118 Safety, Health & Environment Practice Use Index by Respondent Type................................................................................................ 119
Figure 28:
Automation / Integration Technology Practice Use Index by Respondent Type............................................................................ 121
Figure 29: Figure 30: Figure 31: Figure 32: Figure 33: Figure 34: Figure 35: Figure 36: Figure 37: Figure 38: Figure 39:
Owner Project Cost Growth Comparison....................................... 129 Owner Change Cost Factor Comparison........................................ 130 Owner Pre-Project Planning Practice Use Comparison ................. 131 Contractor Project Budget Factor Comparison .............................. 133 Contractor Change Cost Factor Comparison ................................. 134 Contractor Team Building Practice Use Comparison .................... 135 Standardization and Normal Transformation of Sample Data ....... 145 Owner Project Cost Growth vs. Design Practice Use Scatter Plot. 147 Contractor Scatter Plot by Company.............................................. 153 Scree plot for Practice Use Indices Factor Analysis ...................... 165 Owner Project Performance Index vs. Combined Practice Use Index Scatter Plot ........................................................................... 170
xvi
Figure 40:
Contractor Project Performance Index vs. Combined Practice Use Index Scatter Plot .................................................................... 172
xvii
Chapter 1 Introduction
The construction industry has a major impact on the overall United States economy. Statistical data show that the industry represents about 4.6 percent of
the United States gross domestic product (GDP) in from 2001 to 2004 (Strassner and Howells III 2005). The number is substantially higher if other
construction-related industries, such as metal fabrication, industrial machinery and equipment production, wood products, and the furniture industry are included. Because the business environment in industry is highly competitive, improving capital effectiveness is imperative to business success. It is conservatively estimated that 40-50 percent of all construction industry capital budgets are now spent on smaller projects, therefore successful small project execution is especially important (CII 2001a). These small /
special projects are treated differently across the industry and lack a common definition. effectiveness. Many companies deliver small projects with reduced capital The main factors for this include a more compressed project life
cycle combined with less management support due to low visibility. Minimal research has investigated small project execution. Providing tailored practices for small project execution and the validation of their value will contribute to the overall effectiveness of the construction industry. What is a small project? The industry lacks a widely accepted consensus
on the definition of small projects. Furthermore, little research has focused on a 1
quantitative methodology to assess small projects.
Therefore, this research seeks
to develop a common definition as well as a method to evaluate the performance and implementation of project management practices for small capital projects.
1.1. THE CONSTRUCTION INDUSTRY INSTITUTE The Construction Industry Institute (CII) was established in 1983 at The University of Texas at Austin to improve the cost effectiveness of the construction industry. CII is a consortium of leading owners, engineering and construction contractors, suppliers, and academia. It is the recognized principal construction
industry forum for improving the business effectiveness and sustainability of capital facilities and increasing the business success of member organizations. The mission of the Construction Industry Institute is to add value for members through CII research, and related initiatives such as benchmarking, and industry alliances. With increased member participation in CII activities, a
global competitive advantage is realized through active involvement and the effective use of CII research findings, including CII Best Practices. Currently, CIIs membership consists of 89 organizations, including 46 owners and 43 contractors. Members participate in CII committees and
initiatives including research, product implementation, education, benchmarking, knowledge management, breakthrough, and globalization. Since its inception, CII has identified, completed, and reported on numerous research projects based on the joint effort of industry participants and academia. 2 This research effort has
resulted in more than seven hundred CII publications now organized under 14 Knowledge Areas: (1) Front End Planning, (2) Design, (3) Procurement, (4) Construction, (5) Startup and Operation, (6) People, (7) Organization, (8) Project Processes, (9) Project Controls, (10) Contracts, (11) Safety, Health, and Environment, (12) Information Management and Technology Systems, (13) Globalization Issues, and (14) Security.
1.1.1. Benchmarking & Metrics Program The Benchmarking & Metrics (BM&M) Committee was formed in 1994 as an ad hoc committee. It became a standing committee of the Construction
Industry Institute (CII) in 1996, due to an increasing need to provide member companies reliable benchmarking services. As of July 2003, the committee was composed of six teams: (1) Analysis, (2) Questionnaire, (3) Marketing, (4) Productivity Metrics, (5) Small Projects, and (6) Online Training. The
committee members were selected to participate from CII member companies representing both owner and contractor. program include (BMM 2002): Provide the industry with a common set of metrics definitions. Provide the industry with project performance norms. Measure the level of use of selected best practices. Quantify the value of implementing CII recommended best practices. Provide participating companies tools for self-analysis. 3 The objectives of the CII BM&M
Facilitate the development and sharing of benchmarking knowledge within the construction industry.
Provide a credible and member accessible database that is efficient in terms of resources required for data submission, analysis, and the reporting of findings. The CII BM&M database currently contains 1,415 capital projects valued
at over $65 billion in total installed cost.
A notable change in the database in After several years in which
recent years is the trend toward smaller projects.
the size of projects continued to decline due to reasons including reduction in capital programs and more emphasis on modernizations and additions for both owners and contractors (BMM 2002), a strong interest developed to investigate the need for small projects questionnaire. The Small Projects Team was formed as a task team of the BM&M committee in April, 2002. Members of the team
include industry representatives from different owners and contractors, as well as academic researchers. Table 1 lists the companies which were represented in the Small Projects Team:
4
Table 1:
Composition of Small Project Team
Participating Companies/Organizations Construction Industry Institute DOFASCO General Motors GlaxoSmithKline Jacobs Engineering Johnson Control The University of Texas at Austin U.S. General Services Administration
The charter of the team was to: Establish a common definition for small projects. Revise and adapt the existing CII BM&M Questionnaire for small projects. Build on existing research for small projects. Incorporate best practices appropriate for small projects into the questionnaire. Expedite questionnaire development. The small project team met regularly from April 2002 to July 2003 in a concentrated effort and each member provided valuable input in questionnaire development. The composition of the small project task team provided both
diversity among industry practitioners and a balance with academic researchers. The author of this research, Lilin Liang, participated in the teams effort beginning in August 2002. During this time frame, the author conducted a
literature review, developed the research objectives and hypothesis, participated in drafting of the questionnaire, and ultimately collected and analyzed the data.
5
1.2. PROBLEM STATEMENT The growing trend toward smaller capital projects within the engineering and construction industry has many implications for project management. Organizations executing predominately small projects typically handle more numbers of estimates, procurements, and subcontracts. Tailoring large project practices to manage smaller projects is often less effective. Research seeking solutions to better project management for small projects has progressed in the past decade. However, it is evident from the literature review that there is a lack
of both a widely accepted definition and proven management practices for small projects. In addition, the industry lacks a method to evaluate project performance and the implementation of project management practices for small projects. There has also been little research done to evaluate small project programs using empirical data with comparisons made to larger capital projects. Analysis of similarities and differences between small and large projects performance and characteristics will be beneficial to practitioners as well as contribute to the body of knowledge within the engineering and construction industry. Based on the
literature review, the problem statement of this research is as follows: A need exists to identify and develop a common definition for small projects based on empirical data. There also exists a need to develop a method to evaluate project performance and the implementation of project management practices for small projects.
6
1.3. RESEARCH OBJECTIVES The main objectives of this research is to (1) provide a method to evaluate small project performance, (2) develop a method to evaluate the implementation of project management practices for small projects, and (3) evaluate the impact and value of suggested project management practices on small project performance. In fulfilling these main objectives, related secondary objectives
are listed as follows: 1. 2. Review existing research on small project management. Provide a consensus definition of a small project for use in this research. 3. Summarize the differences in project performance and characteristics of large and small capital projects, based on empirical evaluation. 4. Provide project norms, i.e., statistical summaries, for project performance and practice use metrics from the data collected for this research. 5. Identify key factors, i.e., project management practices, required for successful small project execution. 6. Provide guidance to project practitioners for improved small project management.
7
1.4. RESEARCH HYPOTHESIS The primary purpose of this research is to develop a method to evaluate small project performance and practice use, and to assess the impact of practice use on project performance such as cost, schedule, change orders and safety. The following hypotheses are developed to meet these research objectives.
H1:
Project
performance
and
the
implementation
of
project
management practices can be measured for small projects. The first hypothesis is formed to prove that a reliable method can be developed to measure and quantify project performances and the implementation of project management practices on small projects. It is also hoped that such a
method would be easy-to-use and would serve as a data collection tool for evaluating small projects.
H2: There are differences in performance and practice use between large and small projects. The second hypothesis is developed to examine the difference in project performance and characteristics between large and small projects. It is widely
believed that differences exist and project owners should manage large and small projects accordingly. Small projects generally have a reduced project life cycle and limited resources, while large projects generally have longer project duration and a better staffed project team. Therefore, small projects are more susceptible to
8
project cost and schedule variation. This hypothesis will be tested by comparing samples from the CII BM&M database, which includes large and small projects.
H3: Better implementation of small project practices correlates with better project performance. The third hypothesis is that a positive correlation exists between small project practice use and performance in small projects. A high practice use score would represent a better definition of scope, better project controls, and commitment to project success. This would, in general, correspond to better
project performance in terms of cost, schedule, change orders and safety, and therefore reduce risks and improve capital effectiveness. The hypothesis will be
tested by conducting statistical analysis of the CII Benchmarking & Metrics Small Projects Database. The research will continue to provide guidance to industry
practitioners on how to manage small projects.
9
1.5. RESEARCH SCOPE This study focuses on capital projects completed within the past two years which fit the definition of a small project. The definition, based on literature
review, analysis of the CII BM&M database, and consensus from industry experts, is listed below. Detailed discussion regarding development of the small
project definition is presented later in Section 2.2. Small projects, defined for this research, should have at least one or more of the following characteristics: Total installed cost is between $100K and $5M Any duration of 14 months or less Any number of site work-hours up to 100,000 The project does not require full-time project management resources or a significant percentage of company resources Any level of complexity Any type of project including maintenance and expense projects
Data from this research is collected using the CII Benchmarking & Metrics online system (BMM 2004). Participants are required to have sufficient
knowledge of the benchmarking system by either completing CII Benchmarking Associates Training or an equivalent training conducted internally by trained company Benchmarking Associates. The data are collected through the questionnaire developed during this research investigation. Analysis is limited by the contents of the questionnaire, 10
which contains leading performance indicators such as cost, schedule, change orders, and safety. Questionnaire data regarding the implementation of project
management practices on small projects are also collected. Other performance indicators such as quality and productivity are not included in this research and may require a separate effort for comprehensive analysis.
1.6. ORGANIZATION OF DISSERTATION This dissertation consists of seven chapters and a set of appendices that contain the Small Project Questionnaire and detailed statistical results. Chapter Two provides a background review of previous studies in the context of benchmarking, small project definition, small project management, success factors, and practice use correlations. Chapter Three illustrates the
research methodology including questionnaire development, data collection, and statistical techniques employed for hypothesis testing. the Small Project Questionnaire in detail. provides a data summary. Chapter Four describes
Chapter Five defines the metrics and
Chapter Six provides discussion regarding data
relationships, including (1) the differences between small and large projects, and (2) correlations between practice use and project performance. Chapter Seven
summarizes the research conclusions and contributions, provides industry recommendations, and addresses future research opportunities.
11
Chapter 2 Background
2.1. BENCHMARKING IN CONSTRUCTION INDUSTRY Benchmarking is defined as a continuous, systematic process for evaluating the products, services, and work processes of organizations as compared with recognized best practices for the purpose of organizational improvement (Spendolini 1992). The Construction Industry Institute has a similar definition: a systematic process of measuring ones performance against recognized leaders for the purpose of determining best practices that lead to superior performance when adopted and utilized (CII 2002). The origins of the competitive benchmarking process are attributed to the Xerox Corporation, which started its practices facing growing competition from Japanese photocopiers, as early as 1979. However, the progress of an industry The nature of the
accepted benchmarking framework has been slow and limited.
construction industry has caused a tremendous barrier in implementing benchmarking practices among the industry (Lee et al. 2005; Zairi 1992). In 1995, Lema and Price reviewed various literatures and developed a conceptual framework and methodology for benchmarking. Fisher et. al. (1995)
conducted similar research and concluded that, the construction industry had neither developed a benchmarking standard, nor did it have an organization for collecting benchmarking data. 12
In 1997, Hudson published a dissertation presenting the governing philosophy of a benchmarking system for construction projects, and a framework of metrics with a data collection instrument. His work marked the inception of Hudson also suggested
the Benchmarking & Metrics Program at CII.
developing some form of electronic questionnaire to increase usability and to ease the burden of data validation. By the turn of the millennium, it was concluded
that a benchmarking procedure for diverse construction projects was attainable, as suggested by Garnett and Pickrell (2000) and Ramirez et al. (2004). In 1999, the CII Benchmarking and Metrics Program released an electronic version of the questionnaire, which enabled data collection be done continuously. The system later became fully web-based with the incorporation
of real-time reporting of benchmarking norms (Lee et al. 2005).
2.2. SMALL PROJECT DEFINITION It is difficult to find a consensus agreement on the definition of a small project. The Manual for Special Project Management (CII 1991), provides an It concluded that there are a number of As a result, it In light of this,
early definition for small projects.
parameters and classifications as well as differing points-of-view. is difficult to summarize the characteristics of a small project.
CII documented that deciding whether a project is small is an intuitive decision which reflects the companys size, type of work pursued, current work volume, and management approach. 13
Griffith and Headley (1998) explained small projects as follows: Cost of administration expressed as a proportion of the work itself can be much greater than larger projects Short duration (one to three months) Higher uncertainty Limited formal documentation Some small projects have considerable diversity in size, value and complexity and are procured and managed using formal systems Dunston and Reed (2000) considered potential candidates for small projects being those with the following characteristics: Repetitive/routine work Simple/uncomplicated construction process Renovations/remodeling/upgrades Total project costs less than $1,000,000 Maintenance Projects More recent research documented in CIIs Small Project Toolkit (CII 2001b) defined a cost range for small projects of $100,000 to $2,000,000 in total installed cost and having the following characteristics: Less staff and generally managed as part of a program Less formal controls Higher project contingency More standardized process and use of checklists, etc.
14
Rather than relying on the above definitions for this research.
Analyses
were performed using CII BM&M data to determine parameter breakpoints for small projects. First the team, consisting of a panel of eight industry experts
from seven organizations, agreed that a total installed cost of $5 million was suitable as an upper bound. Data were then used to determine the appropriate
duration and number of work-hours correlating to this total installed cost. The team concluded, based on the literature review, that small projects are those not requiring full-time project management resources and that those projects could exhibit any level of complexity. Detailed analysis models and explanation of the
method are included in Appendix B. It is concluded that small projects, as defined for this research, should have at least one or more of the following characteristics: Total installed cost is between $100K and $5M Any duration of 14 months or less Any number of site work-hours up to 100,000 The project does not require full-time project management resources or a significant percentage of company resources Any level of complexity Any type of project including maintenance and expense projects
15
2.3. SMALL PROJECT MANAGEMENT Many companies have incorporated standard operational procedures into their management of projects. These time-tested procedures, however, still
focus on projects that have a larger scale. These procedures are oftentimes hard to scale down when adapting for small projects. They are either inadequate
when no formal small project process exists or cumbersome when large project processes and documentation are tailored. These factors can result in reduced capital effectiveness. small projects include: Small Projects are usually managed as a group of projects, sometimes organized formally into a dedicated small project program. The Some of the major issues and concerns of managing such
team members must have the capability of dealing with multiple projects, contracts, and subcontractors. This is different from large
projects where traditionally there is a dedicated project team for only one project (Westney 1985). Due to a tighter schedule and fewer resources, small projects usually are not effectively managed using detailed activity schedules. Smaller projects tend to be controlled using milestone schedules. Therefore, controlling for critical tasks and avoiding schedule delays are essential for successful schedule management (Westney 1985). The compressed project life cycle means there is little room for error. A large project may have the time for project teams to correct and fix process problems. The duration of a small project can be as short as 16
a few weeks (Westney 1985).
These projects are also likely to have
experience gaps, where there is a lack of documentation of lessons learned as project teams tend to move on quickly to other projects. CII publications, especially Factors Impacting Small Capital Project Execution, (CII 2001a) have been helpful in locating existing research on the execution of small projects. This report identifies various factors which can be
grouped into the categories of front-end planning, design, procurement, construction, start-up and operation, people, organization, project processes, project controls, and contracting.
2.4. PROJECT SUCCESS FACTORS There has been much research investigating the elements of projects success. This includes the evaluation and measurement of project success
through key performance indicators (KPIs) and critical success factors (CSFs). Significant findings are listed below. Ashley et. al, (1987), defined project success as better results in project outcomes including cost, schedule, quality, safety, functionality, and participant satisfaction. Sanvido et. al., (1992), stated that success for a given project participant is defined as the degree to which project goals and expectations are met. These goals and expectations may include technical, financial, educational, social, and professional aspects and may be different for different project participants. 17
Chua et. al, (2000), investigated critical success factors (CSF) distinguished by three major categories, (1) budget performance, (2) schedule performance, and (3) quality performance. performance include: Project Manager competency Budget updates PM commitment and involvement Design complete at construction start Formal communication during design PM authority Constructability program Formal communication during construction Construction control meetings Design control meetings The major factors for schedule performance include: PM competency PM commitment and involvement Schedule updates Construction control meetings Capability of contractor key personnel Site inspections Formal communication during construction Constructability program 18 Major factors for budget
PM authority Competency of contractor proposed team Cox et. al, (2003) investigated key performance indicators for construction projects. The research concluded that the most commonly used indicators which
were highly significant to industry participants included: Quality Control, On-Time Completion, Cost, Safety, $/Unit, and Units/Man-Hours. Chan et. al, (2004) put together a conceptual framework on critical success factors which were categorized as: (1) General Project Factors, (2) Procurement procedures, (3) Project Management Actions, (4) Human-related Factors, and (5) External Environment Factors.
2.5. PROJECT PERFORMANCE AND PRACTICE USE CORRELATIONS The relationship between project performance and the factors that impact them has been extensively explored. Researchers have employed different
statistical techniques to support this analysis. Some of the research is listed below. Jaselskis and Ashley (1991) studied the impacts of various management practices on different project objectives using the logit regression model. Their
research studied the relationship between project outcomes, including (1) overall project success; (2) schedule success; and (3) budget success and other factors such as (1) project team, (2) project planning, and (3) control effects. They 19
concluded that each factor has a different degree of impact on the project outcome. Kaming et. al., (1997) examined factors influencing construction time and cost overruns on high-rise projects in Indonesia using a questionnaire survey followed by the principal component factor analysis technique. This research indicated that the factors influencing delays and cost overruns can be different. In the case of delays, the predominant factors are design changes, poor labor productivity, inadequate planning, and resource shortages, while the more dominate factors influencing cost overruns are inflation, inaccurate materials estimating, and the degree of project complexity. In his dissertation, Hudson (1997) documented the history of the CII benchmarking program, the development of project performance and best practices metrics, and the benchmarking survey instrument for large projects. He also performed studies examining the relationships between project performance outcomes and CII best practices including pre-project planning, team building, and constructability using simple linear regression. Morrow (1998), in his dissertation, provided extensive information on the effects between best practices and project cost performance. He used ANOVA
and simple linear regression techniques to study the relationship among practice uses. He also quantified the relative importance among certain practice uses
using a combined best practice use index. In addition to his application of the combined practice use index, he also considered the importance of project
20
environment factors in quantifying the correlations between best practice use and project performance. Lee (2001) first used discriminate function analysis (DFA) to further quantify the relative importance of best practices to project performance. In addition, Lee also developed a DFA software tool that allows predictive classification of project performance before project start (Lee et. al. 2004). The
DFA program can be helpful in assisting industry assess project performance early on and will likely improve their performance.
2.6. LITERATURE REVIEW SUMMARY The literature review provides background for this research by showing previous studies focused on (1) benchmarking theory and practices in the construction industry, (2) development of a small project definition, (3) small project management, (4) project success factors, and (5) relationship of project performance and practice use. Different definitions of small projects have been previously suggested. Past research shows that industry group, company size, and experience influence perceptions of small projects. for the term small project. As such, the industry lacks a common definition The small project definition, as discussed in
Section 2.2, is based on not only existing literature, but also correlations drawn on empirical data from the CII Benchmarking & Metrics Database.
21
Commonly employed small project practices are usually a subset of large project practices or are pieces of them tailored specifically to fit project needs. Based on the diversity found in the literature and practice, the industry seems to be short of a widely accepted project management practice for small projects. Project success is measured in a variety of ways, depending on whether the focus is on cost, schedule, quality or other indicators. None of the previous studies, however, provides the impact of practice implementation on project performances specifically for small projects. There has been much research studying the relationship of large project performance and practice use. The correlation of more practice use leading to
better project performance for large projects has been established and well documented. It is important to note that none of the previous studies have evaluated small projects using empirical data. Most of the research was conducted by qualitative surveys at the organization level rather than at the project level.
22
Chapter 3 Research Methodology
The research methodology is laid out in Figure 1, outlining research events that help to ensure the completion of the research objective. descriptions of the research methodology follows. More detailed
Figure 1:
Research Methodology 23
3.1. QUESTIONNAIRE DEVELOPMENT The Small Project Questionnaire was developed in three stages to ensure the final product is suitable for general use across different industry sectors. The
questionnaire development included a review of current research, modification of the existing Benchmarking & Metrics Questionnaire for Large Projects (Version 7), and intensive research meetings with industry experts. The author of this dissertation, participated in the drafting of the questionnaire and was responsible for facilitating the questionnaire development process. The majority of the work done initially during questionnaire development was in the project performance section. In developing the questionnaire, the goals were to be concise and include only essential data for evaluating project performance. Project information such as cost, schedule, changes, and safety
were identified as critical components to be included. Next, recommended project management process questions were developed. Many of the questions were adapted from the Small Project Toolkit (CII 2001b). Questionnaire contents were based on related research and The questions were developed into a checklist
combined with expert opinions.
format to facilitate their use and to lessen the burden on questionnaire respondents. The Likert scale was selected to measure the strength of agreement with practice use statements according to a five-point scale that ranges from (1) Strongly disagree, (2) Disagree, (3) Neutral, (4) Agree, and (5) Strongly agree. The Likert scale was named after Rensis Likert, who invented the scale in 1932. 24
The Likert scale is often used to measure attitudes, preferences, and subjective reactions. Likert scales help get at the emotional and preferential responses people have to the design. Finally, the team reviewed and modified the questionnaire in order to reduce the number of questions by eliminating redundancy and to identify any final additions that would contribute to the overall body of small project management practices. 3.2. QUESTIONNAIRE PILOT TESTING Pilot testing started with the final stage of questionnaire development. The main goal for the pilot testing was to receive initial feedback from a selected group of project participants. The feedback was then considered in the final drafting of the Small Project Questionnaire. The questionnaire was subsequently coded for online submission of data by the author. The pilot Small Project Questionnaire version 1.1 included 158 questions, 77 in the general information and project performance section, and 81 in the practice use section. All the questions in the practice use section were weighed equally. Once sufficient data are available, and a preliminary analysis is
performed, the algorithm practice use weighting should be revisited. Twenty pilot projects were collected with substantially complete data. The online questionnaire was modified using team input to include programming fixes, improved presentation, and the addition of user-friendly features such as online help and glossary.
25
3.3. DATA COLLECTION AND VALIDATION To enhance the functionality and visibility of the questionnaire, the completed Small Project Questionnaire was then integrated into the CII Benchmarking & Metrics website (BMM 2004) for data collection. Data for CII
BM&M are collected at http://cii-benchmarking.org, a secure site offering confidential project input with real-time feedback. Online data collection
provides flexibility for project participants to enter data, it is recommended that data be input during project execution to ensure accuracy and to access project participant knowledge before the team disperses. After the data are collected, a manual data validation process is enforced to check for consistency. If an inconsistent response is suspected, the response is flagged and then communicated with the respondent for error checking. The
goal of the validation process is to increase data quality by eliminating errors and omissions in the data collection. Each participating company is assigned an Account Manager to assist in validation of data. Account Managers are graduate students, the author included, Account
whose research involves various benchmarking areas of study.
Managers work with company representatives to ensure data are as correct and complete as possible.
26
3.4. DATA ANALYSIS TECHNIQUES A descriptive statistical analysis was performed by the author to analyze the major metrics by different respondent type, industry group, and project characteristics. The statistical results represent the distribution of industry The information is useful to industry
performance in a particular subset.
practitioners when comparing their project performance to other projects in the same group. The analysis also reveals the best-in-class performance, which
represents best performance reached in a particular subset. The analysis includes statistical techniques to quantify and test the difference between large and small projects. Analysis results establish the
second research hypothesis: there are differences in performance and practice use between large and small projects. In addition, statistical procedures are used to assess the correlation of practice implementation on project performances in an attempt to seek evidence to support the third research hypothesis: better implementation of small project practices correlates with better project performance. Depending on the number
of comparison groups and distribution of sample variances, either a standard T-test or Analysis of Variance (ANOVA) technique is performed (Agresti and Finlay 1999). Regression techniques are also employed to study the
relationships of different practices and project performance if sufficient data exists (Bobko 2001).
27
3.4.1. Box and Whisker Diagrams The box and whisker diagram, also called a box plot, portrays the range and the quartiles of the data, and possibly some outliers. The central portion, or
the box, consists of 50 percent of the data, from the first quartile (25th percentile) to the third quartile (75th percentile). inter-quartile range (IQR). This range is also referred to as the
The median is drawn as a horizontal line in the box. Plotting both
The mean, according to graphical preferences, can also be plotted.
the median and the mean offers an efficient way to indicate the central tendency. The whiskers extend away from the box, indicating the range of the data that is not considered an outlier (Agresti and Finlay 1999). The end points of whiskers represent the last data observation that falls within the 1.5 IQR fence. A sample box plot is included in Figure 2.
28
Sample Box and Whisker Diagram
Outlier Symbol Las t Obs ervation below (Q3 + 1.5IQR)
Third Quartile (Q3) Median
Mean
Firs t Quartile (Q1) Las t Obs ervation above (Q1 - 1.5IQR)
Figure 2:
Sample Box and Whisker Diagram
Outliers are values that fall 1.5 IQR beyond the third quartile or 1.5 IQR below the first quartile, as indicated in Figure 2. According to the range of variation, extreme outliers are defined as those observations locating beyond 3.0 IQR from the box. Values between 1.5 IQR and 3.0 IQR from the box are
regarded as normal outliers. The box plot is an efficient way to present both the central tendency and range, which makes it extremely useful in comparing data among groups. The
data presented can help users compare the median, mean, and variance between groups, draw conclusions among the differences, and contribute to decision making (Agresti and Finlay 1999).
29
3.4.2. Linear Regression Analysis The simple linear regression model: E (Y ) = + X , explains the
relationship between the explanatory (independent) variable X and the mean of the response (dependent) variable Y. The intercept, , is the expected value of Y when X is zero. The regression coefficient, , represents the change in the expected value of Y with one unit of change in X. The model is referred to as the bi-variate model, since it contains two variables. environment, the above model can be generalized to E (Y ) = + 1 X 1 + 2 X 2 + ... + k X k Or
Y j = + 1 X 1 j + 2 X 2 j + ... + k X kj + e j The above is called a multiple regression equation, where e j indicates the model residual, or error. When k=1, the equation reduces to the original bi-variate model; when k=2, the equation represents a plane in a three dimensional space. For k>2, the equation represents a (k+1)-dimensional space, which is more difficult to visualize. These descriptions are based on Statistical Methods for the Social Sciences, by Alan Agresti and Barbara Finlay (1999), including the goodness of fit, statistical inferences for the regression model and basic model assumptions. Each subject in the equation is fit using the Least Squares Method in minimizing the error term between the observed and predicted Y. proportional reduction in error from using the prediction The
In a multivariate
equation
30
Y = + 1 X 1 + 2 X 2 + ... + k X k instead of the mean, Y , to predict Y is call
^
the coefficient of multiple determination. This is defined as:
R2 =
(Y Y ) (Y Y ) (Y Y )
2 2
2
=
TSS SSE TSS
The R 2 measures the proportion of the total variation in Y that is explained by the simultaneous predictive power of all the explanatory variables. R 2 falls between 0 and 1. A greater R 2 indicates a better model fit. equals to 1 only when all the residuals are zero, indicating a perfect fit. The statistical inferences checks whether the explanatory variables are statistically related to Y, i.e., whether the explanatory variables collectively have a statistically significant effect on the response variable Y. The test is conducted by checking the null hypothesis H 0 : 1 = 2 = ... = k = 0 This states that the mean of Y does not depend on the values of X1, X2, , Xk, i.e., if all the partial regression coefficients equals to 0, then Y is statistically independent of all k explanatory variables. The alternative hypothesis is H a : At least one i 0 This states that at least one explanatory variable is related to Y, controlling for other variables. The test statistic equals: R2 / k F= (1 R 2 ) /[ n (k + 1)] R2
31
The sampling distribution of this statistic is the F distribution, which assumes nonnegative values and is skewed to the right. Statistical packages report the actual p-value for each calculated F statistic. The p-value is the
probability that your sample could have been drawn from the population being tested given the assumption that the null hypothesis is true. A p-value of 0.05, for example, indicates that you would have only a 5 percent chance of drawing the sample being tested if the null hypothesis was actually true. More
specifically, the p-value of a statistical significance test represents the probability of obtaining values of the test statistic that are equal to or greater in magnitude than the observed test statistic. A p-value of 0.05 or lower, at a significance level of 0.05, is generally used as the criteria indicating statistical significance. The appropriate interpretation of regression analysis is dependent on how well certain underlying assumptions are made. Although regression can be fairly robust, the correct interpretation and usage of the regression model subscribes to these basic model assumptions: The sample is randomly selected. Independence (i.e., the error associated with each observation is independent of every other error value). e The population mean of j (error) is zero. ej e 2 . For any value of X, the The population variance of is error variance should be constant and have a normal distribution. The assumption of equal error variance is labeled homoscedasticity.
32
There is no perfect collinearity no explanatory variable is perfectly correlated with one or more of the explanatory variables.
3.4.3. One-Way Analysis of Variance
One of the most commonly used analyses refers to comparing the mean responses of several groups for quantitative response variables. The method is called analysis of variance (ANOVA). There are a few underlying assumptions governing the test. Assuming that there are a total of g groups, the assumptions are as follows (Agresti and Finlay 1999): The population distributions on the response variable for the g groups are normal. The standard deviations of the population distributions for the g groups are equal. Independent random samples are selected from the g populations. The null hypothesis for the test is H 0 : 1 = 2 = ... = g . Where
i
denotes the population mean for group i. The test statistic for H0 indicates the ratio of the between estimate to the within estimate,
Between Estimate BSS /( g 1) F= = = Within Estimate WSS /( N g )
n (Y Y ) /( g 1) (n 1)s /( N g )
2
i
i
i
2
i
i
i
where
N : Total Number of Samples Y : Overall Mean ni : Number of observations in group i 33
si : Sample Variance for group i
2
Yi : Mean for group i The p-value is reported in statistical packages indicating the right-hand tail probability. The larger the F statistic, the smaller the p-value. Statistical
significance of the test is generally established by rejecting the null hypothesis at a p-value lower than 0.05.
3.5. LIMITATION AND BIASES
Crosson (1994) makes several recommendations in a book summarizing the potential difficulties with conducting and interpreting survey research. Crosson notes, As a general rule, the less information that is available about the way a poll was conducted, the less it can be trusted. Web data collection is the primary source of data being collected by this research, thus the data is not polled randomly from the true population. The subjects that did not participate in the survey may result in undercoverage (Agresti and Finlay 1999). In addition, the questions in the practice use section survey subjective responses about project management practices for small projects. The potential for response bias is present if the respondents overcompliment the actual situation when they considered their response may be unacceptable (Agresti and Finlay 1999). This is considered a form of the social desirability (SD) bias (Crowne and Marlowe 1964). 34
CII conducts Benchmarking Associates Training three times per year to company participants interested in the Benchmarking & Metrics Program. The purpose of the training is to enhance data quality by increasing knowledge and awareness to the functionality of the benchmarking system and various metric definitions. Participants who enter the data through the online system must
complete either the CII Benchmarking Associates Training or attend equivalent training hosted by the companys Benchmarking Associate. Additional data
quality checks are included in the data validation process after the data are submitted for analysis. A requirement of benchmarking with CII is that
companies comply with a code of conduct agreeing to submit representative project and accurate data.
35
Chapter 4 The Small Project Questionnaire Measuring Project Performance, Practice Use and Other Factors
This chapter outlines the Small Project Questionnaire. The author, along with input from the CII BM&M Small Project Team and the literature, developed the questionnaire the descriptions given in this chapter. The questionnaire is divided into three major sections: (1) general information, (2) performance and (3) practice use. A more detailed description of the contents in each section is presented below. Appendix A.
4.1. GENERAL INFORMATION SECTION
The complete questionnaire is available for reference in
The general information section records the project characteristics including project location, industry group, project nature (grass roots, addition, modernization, or maintenance), delivery system, complexity, etc. These factors are used to categorize projects into various slices for comparison purposes. A project scope question allows each respondent to provide a narrative description. The feedback from project scope information can be utilized to further classify projects. The final part of this section concerns project participants. For a respondent in an owner organization, information about each of the primary project functions is gathered. For each project function identified, owners are asked to estimate a percent participation contributed from either the owner itself, 36
alliance contractors, or non-alliance contractors. The percent allocated for each project function should total to 100 percent. Contractors, however, are only surveyed on their scope of work. Unlike owners, they only need to report their
effort for the specific project functions in which they participated. The data collected here show the number of different parties involved and whether alliances are heavily used. The information can also be useful to categorize the complexity of project. After the initial release of the questionnaire, two new questions were added to survey, one to capture the primary project driver, and the other turnaround information. These questions were suggested by active participants and were endorsed by the Small Project Team. A pull-down list of options is used to identify the main project driver. The options currently listed includes: (1) cost, (2) schedule, (3) meeting project specifications, (4) regulatory requirements, (5) production capacity, and (6) others. The primary project driver can be useful in constructing weighting criteria to determine overall project success. The turnaround question asks participants to identify the percent of construction work executed during a plant turnaround. Turnaround is a term interchangeable with shutdown or outage depending on industry groups. Construction performance (cost, schedule, and safety) during project turnarounds may be impacted by schedule demands of the outage. The data collected in this section can aid future analysis to determine the full impact of project turnaround.
37
4.2. PROJECT PERFORMANCE SECTION
A section of the Small Project Questionnaire is dedicated to gathering project performance information for cost, schedule, changes, and safety. For project cost and schedule, baseline and actual figures are collected by phase. A tally of phase project changes is captured as well as the net cost impact, and the net schedule impact of changes generated from the design or construction phase. Accident data and work-hours are recorded according to OSHA standards to calculate safety metrics to include the total recordable incidence rate (TRIR) and days away, restricted, or transfer (DART) rate.
4.2.1. Project Cost Performance
The cost section gathers the budget and actual costs by phase. These phases include pre-project planning, detailed design, procurement, construction, and start-up. The demolition phase, originally included in the large project
questionnaire, was cut out of the Small Project Questionnaire since the majority of small projects do not have a significant amount demolition work. Flexibility is built in to allow participants to enter only the total project budget and costs if respondents do not want to provide the detailed phase costs or if it is unavailable. Project contingency is recorded as well. In the Large Project
Questionnaire, project contingency is surveyed by phase, but is simplified here to a single rollup question in order to reduce the burden of input.
38
The cost data is utilized to calculate cost performance metrics. These metrics include: (1) Project Cost Growth, (2) Delta Cost Growth, (3) Project Budget Factor, and (4) Delta Budget Factor. presentation of data will be discussed in Section 5.2. Metrics definitions and
4.2.2. Project Schedule Performance
In the schedule section, participants report the planned and actual start and finish dates for the overall project schedule. Collecting the start and finish dates not only provides a means for calculating schedule duration, but also serves as a time stamp to indicate when the project was executed (constructed). Additionally, respondents enter planned and actual duration, in calendar days, for each phase. Calendar days are specified to provide a point of
comparison with the overall project start and stop dates. If respondents report the durations in work days, any calculations made with the overall schedule duration will be flawed. The definition and calculation of the phase duration factor is discussed later in Section 5.2.5.2. For owners, questions regarding the amount of detailed design completed at time of project authorization and construction start are also asked. These two questions originate from the Large Project Questionnaire. Industry participants frequently inquire with CII BM&M for the norms of percent detailed design done for various industry sectors. These questions may also allow investigation of the relationship of percent design complete with other project factors. 39
4.2.3. Project Change Performance
Project change performance collects the information regarding project change orders, including the number of changes initiated, the net cost impact of the changes, and the net schedule impact of changes. In the Small Project
Questionnaire, changes are tracked for both the design and construction phase, depending on when the change order is initiated. provide this detail, a total may be input instead. In the large project questionnaire, participants are asked to separate change orders into project development changes and scope changes. Project development changes, as defined in the Large Project Questionnaire, are those changes required to execute the original scope of work or to obtain original process basis, including: (1) unforeseen site conditions that require a change in design/construction methods, (2) changes required due to errors and omissions, (3) acceleration, (4) change in owner preferences, (5) additional equipment or processes required to obtain original planned throughput, and (6) operability or maintainability changes. Scope changes are defined as changes in the base If the participant cannot
scope of work or process basis. Since small projects usually operate on a much compressed timeline, it would be more difficult for participants to determine the type of change and to enter those changes into detailed phases (pre-project planning, detailed, design, procurement, construction, and start-up). As a result, the questionnaire was
streamlined and participants are asked to report changes from either design or
40
construction phase. To provide flexibility, questionnaire participants can report the total changes if phase data is unavailable.
4.2.4. Project Safety Performance
The safety performance section collects data necessary for the computation of Occupational Safety & Health Administration (OSHA) metrics. The questionnaire uses a format where data can be readily extracted from the OSHA 300 log, instituted on January 31, 2002. The questionnaire asks
participants to enter the total OSHA number of recordable incident cases, including injures, illnesses, fatalities, transfers, and restrictions. Participants are then asked to breakdown the recordable cases into the numbers of (1) injuries, (2) illnesses, and (3) fatalities, respectively. Furthermore, the questionnaire asks respondents to report the total number of OSHA Days Away, Restricted, or Transfer (DART) cases, broken down in (1) Days Away, (2) Restricted, and (3) Transfer cases. Total site work-hours are also collected for the calculation of incidence rates. Additional information collected in this section includes: (1) total number of near misses, (2) percentage of overtime hours, and (3) hours in a normal work week. A definition of near misses adapted from Heberle, 1998, is provided. A near miss is an incident that does not result in injury, but may cause property damage. If, for example, an employee had been in a slightly different position or place, or the equipment or product placement had been to the left or right, serious 41
injury and/or damages could have resulted. Much depends on sheer luck and circumstance. Data provided for percentage of overtime hours and the normal week hours are collected for future analysis that may include the overtime impact on project safety performance. The safety section in the Small Project Questionnaire is the only project performance section that is adapted completely from the Large Project Questionnaire. Safety is a priority in the industry and hence a leading project indicator. It was determined by the team that all information collected was useful and would not increase the burden to questionnaire respondents.
4.3. PRACTICE USE QUESTIONS
The practice use section is categorized into ten areas including: (1) front-end planning, (2) design, (3) procurement, (4) construction, (5) start-up and commissioning, (6) organization, (7) project processes, (8) project controls, (9) safety, and (10) automation / integration technology. This structure generally aligns with the categories defined in the Small Project Toolkit and the CII Knowledge Structure. The questions in the practice use section are provided in a checklist format to evaluate the implementation of project management practices in an effective and consistent manner. Most practice use questions were adopted from the
Small Project Toolkit with modifications from the team of industry experts. The following sections describe each practice use category and present a description 42
of each question, along with the original research from which it was adopted. Practice use questions are formatted to allow a five-point Likert scale of responses from strongly disagree (no practice implementation) to strongly agree (full implementation).
4.3.1. Front-End Planning Practice
Performing effective front-end planning is challenging for small projects due to the compressed timeframe and competing project resources. Successful projects maximize their chances of success by obtaining early and effective input from all stakeholders and ensuring that project objectives and concepts are adequately conveyed. The goal of this section is to measure the overall Ten recommended practice
front-end planning effort on a specific project. questions are included.
Question A: Project objectives / concepts were adequately conveyed to the Front-End Planning team. The Small Project Toolkit (CII 2001b) states
that the objective of the front-end planning effort is to finalize, fix, and communicate the project scope. The intent of this question is to measure how well the project objectives are conveyed at the beginning of a project.
Question B: Front-End Planning & Estimating were funded from a general program fund or other non-project sources. Planning and estimating
activities for projects can be funded either from a general program fund or from the specific project. Funding from a general program fund or other non-project 43
sources is more dependable and consistent.
Having consistent, dependable
funding sources that a team can count on from one quarter to the next allows the team to dedicate resources and plan the work in a logical, effective manner (CII 2001a).
Question C: The Front-End Planning team (including contractors and end users) was both integrated and aligned.
The integral piece of project
alignment is that stakeholders are appropriately represented and that communications are open and effective (CII 1997). This question surveys
whether stakeholders are well represented and whether there is an alignment of objectives. A well-represented planning team will help ensure the quality and completeness of the front-end planning effort.
Question D: Constructability feedback was integrated into Front-End Planning.
Constructability is the effective and timely integration of The benefits of
construction knowledge prior to the start of construction.
constructability have been well documented (CII 1993a and CII 2002a). These benefits include improved project team relationships, minimized rework and better project outcomes.
Question E: Checklists were used to ensure consistency of the Front-End Planning effort.
Specialized project checklists are commonly
viewed as a reminder for execution of major business or project elements. The contents of checklists generally include development of project objectives, project scope definition, business decisions, engineering cost estimating issues, general engineering and construction requirements, etc (CII 1996 and CII 1999a). 44
Question F: A PDRI (CII Project Definition Rating Index) or similar process was used to determine how well the project was defined. PDRI is a tool
for measuring the degree of scope development (CII 1996 and CII 1999a). It provides the project team a positive feedback on the level of scope definition. The tool has been proven to affect project outcomes. The Small Project Toolkit (CII 2001b) suggests implementing a customized version of PDRI as a part of the front-end planning effort.
Question G: Contingency funds were increased as compared to large projects, ideally by three to five percentage points. Since small projects have
less room for budget errors as a percent of total installed cost, the Small Project Toolkit (CII 2001b) suggests that small projects should have a larger contingency as compared to large projects. The Small Project Toolkit suggests an increase of contingency by three to five percent. If the project had more than or less than the recommended percentage (by not answering strongly agree), they receive a lower question score.
Question H: The Front-End Planning team clearly defined the projects priorities such as cost, schedule, and quality.
This question reinforces the
intent of questions A and C. The origin of this question is in the alignment thermometer, a tool for evaluating alignment during pre-project planning (CII 1997).
Question I: Front-End Planning was timely and met schedule requirements.
Part of the success of front-end planning depends on the
45
timeliness of the effort. The project team should be allowed sufficient time to evaluate the information provided by the front-end planning effort.
Question J: The quality of Front-End Planning met project objectives.
It is important to ensure that the project team is satisfied with the quality of the front-end planning effort. This question surveys the completeness of the
front-end planning effort in general. Its addition was suggested by the BM&M Small Project Team.
4.3.2. Design Practice
For small projects, the mission of the design function is to provide the level of design appropriate to the requirements of the project to ensure that the project is executed effectively. Selecting the proper design team and utilizing appropriate design tools are keys to completing small project design successfully. Twelve questions were included in this section to evaluate the overall effectiveness of small project design.
Question A: The scope was frozen before the start of detail design.
Freezing the scope is probably one of the more important aspects of a successful design. Industry practitioners have stressed the importance of working on a fixed project scope, oftentimes a result of a well executed front-end planning effort. It is also easier to allocate project resources on a well scoped project, therefore, it is recommended that the project scope be established early on and that scope creep remain under control (CII 2001a). 46
Question B: The design team (including site knowledgeable personnel) was both integrated and aligned. The design team should have members who
can handle multiple disciplines and be composed of site knowledgeable personnel. It is also important that the team members are integrated and aligned.
Question C: Standardized designs were incorporated into the project.
Standardized work processes have a positive impact on delivering projects with better cost and schedule performance. Organizations using standardized work processes benefit from better project performance (CII 2001a).
Question D: Constructability feedback was integrated into the design phase.
This question reinforces the importance of early constructability
feedback into the planning and design phases, reinforcing the intent of question D in the front-end planning practice (CII 1993a).
Question E: Small project checklists were used to standardize and speed up engineering.
Using a checklist is a way to increase standardization of
processes. It is also a way to increase alignment between project members. Checklists capture major components of essential work. Therefore, using an appropriate checklist can lead to increased productivity and better project performance.
Question F: Design status review meetings were conducted as appropriate. The Small Project Toolkit (CII 2001b) recommends design teams
conduct weekly review meetings. The weekly meetings are typically used to document design progress on all active projects in a portfolio of projects, instead of focusing on one particular project. In contrast to large project meetings, 47
design changes may be discussed over the phone and documented in a short memo or phone conversation report. It is most important that the project team members use appropriate methods to streamline the design meetings and to maximize their effectiveness.
Question G: Design changes were promptly communicated to team members. Small projects have a quick turnaround time, therefore, any delay that
occurs will be detrimental to the project schedule. This makes it essential that the design changes are communicated promptly to other team members, especially members that are affected by the change (CII 2001a).
Question H: Appropriate design controls were used on the project (e.g. budget, schedules, checking, authorizations, and scope changes).
Small
projects entail simpler design controls. The review, check, and approval process should be easier and faster due to the combination of fewer steps, shorter approval routing, and concurrent reviews (CII 2001a). Typically, both small and large projects should have similar infrastructure for design document control. Small projects differ in having less information and less formal methods of communication.
Question I: The engineering and design budget met project objectives. Question J: The engineering and design schedule met project objectives.
Questions I and J stress the importance of controlling for the design budget and schedule. Failure to maintain a fixed design budget and schedule could
indicate excessive project changes, which could be detrimental to overall project performance. 48
Question K: The level of detailed design on this project was adequate (i.e. not under- or over- designed). It is important in the detailed design process
to select the right design deliverables that best fit the small project, without under-designing or over-designing. Establishing the right depth of design also requires correct staffing with knowledgeable personnel.
Question L: The quality of engineering and design effort adequately met project objectives. This question surveys the completeness of the engineering
and design effort in general, similar to question J in the front-end planning section. It is an addition suggested by the BM&M Small Project Team.
4.3.3. Procurement Practice
The procurement of materials and equipment is critical to the success of small projects. Due to the short duration of most small projects, errors and omissions in the procurement process have a magnified impact. It is imperative that items show up on the jobsite when they are needed, where they are needed, and without quality problems. Six questions are designed to evaluate the level of the procurement effort.
Question A: The procurement objectives were communicated to the project team at the beginning of the project.
The procurement objectives
critical to an effective procurement plan begins with clear communication of the owners expectations and philosophies to the engineer and constructor. The
49
procurement objectives further includes general materials management guidelines such as materials receiving, inspection, site storage and logistics.
Question B: Preferred suppliers were used effectively to streamline the procurement process. The Small Project Toolkit (CII 2001b) suggests the use
of preferred suppliers. Preferred suppliers reduce the risk of late deliveries. If project owners pursue only a limited number of small projects (e.g., less than two per year), there may not be a strong potential toward establishing and maintaining a preferred supplier relationship. In this case, sufficient time should be allowed on the procurement schedule to implement a competitive bid process. The
BM&M Small Project Team experts also suggested that using preferred suppliers may ultimately be based on owner philosophies and should be communicated to the project team early.
Question C: Non-alliance contractors were pre-qualified. The key to
a successful small project contracting strategy is to minimize the time and effort required to mobilize a contractor. Mature alliances allow owners to reduce their engineering and construction staff and to use alliance contractors providing the best overall value. If a formal alliance is not in place, the owner should maintain a list of pre-qualified engineering and construction firms. These firms can be selected based on their safety and financial records (CII 1999b and CII 2001a).
Question D: A partnership mentality, as opposed to hard contract incentives, was used on the project. The Small Project Toolkit (CII 2001b)
suggests avoiding the use of contract incentives. The hard incentive dollars tend to divert owner and contractor attention away from the projects at hand and 50
toward contracting terms.
The toolkit further suggests that a partnership
mentality is necessary by all parties for a successful alliance relationship. The BM&M Small Project Team felt that the data collected from this question would help further determine which strategy is used more frequently. Additional data analysis will be needed for further recommendations on wording modification or scoring changes.
Question E: Materials were effectively received, inspected, tracked,
reported, and delivered over the life of the project. This question originates
from the CII Benchmarking & Metrics Large Project Questionnaire. It addresses the importance of the materials management strategy. Materials management is an integrated process for planning and controlling all necessary efforts to make certain that the quality and quantity of materials and equipment are appropriately specified in a timely manner, are obtained at a reasonable cost, and are available when needed (CII 1999b and CII 2002a). The question is designed to evaluate the effectiveness of materials management.
Question F: The owners on-site receiving, warehousing and materials
management systems were effectively used. Since there is normally no specific
receiving/laydown area for a small project, the owners will often allow the constructor to use its facility as the site receiving system. The Small Project
Toolkit (CII 2001b) states that the use of on-site receiving and warehousing is considered a recommended practice. These materials management systems
necessitate capabilities such as (1) tracking of expediting status information, (2) recording supplier performance, (3) showing availability of materials based on 51
schedule, (4) reporting of back order material status, and (6) tracking of bulk materials, engineered material, and tagged items (CII 1999b). The use of
owners facilities and the integration of the materials management system both require early coordination and a clear understanding of available resources, which is a key to procurement success.
4.3.4. Construction Practice
The construction of small projects, in many ways, is more complicated than larger projects. The safety risks, due to working in operating units, can often be greater. Nine recommended practice use questions are included in this section to evaluate the effectiveness of construction planning and management.
Question A: The construction team (including the owner, engineering and procurement) was both integrated and aligned.
The first question Since
evaluates the alignment and integration of the construction team.
construction work on small projects is often pursued at operating facilities, it is important that the constructors follow the owners input and integrate work schedules as appropriate.
Question B: Drawings, site permits, and other required documents were available before starting construction.
Before starting the construction process,
it is imperative that all required documents are in place. Besides drawings and site permits, other required documents include regulatory documents, environmental permits, etc. This question evaluates whether the deliverables 52
from detailed design were in place before starting construction. Questions B through E evaluate the different required elements before start of construction and were aggregated from both the Small Project Toolkit (CII 2001b) and suggestions by the Small Project Team.
Question C: All necessary material, equipment, tools, and work permits were available before starting construction.
This question evaluates the
effectiveness of planning and preparation efforts of the construction process. Thorough checks shall be focused on the required material, equipment, tools, and work permits before starting construction.
Question D: An effective process was used to monitor and control work permits. The Small Project Toolkit suggests that a system be established to
control the work permits for opening lines, energizing and de-energizing circuits, welding, and other operations (CII 2001b). This question is designed to assess the management of work permits.
Question E: Required construction and management personnel were available as needed before starting construction.
This is another question
concerning required elements prior to construction.
Question F: Small project checklists were used to standardize and speed up construction.
Similar questions regarding the use of checklists are also asked Using checklists on project
in front-end planning and design practice use.
processes has a direct correlation with standardization of work processes and increased alignment of team members, both of which have been proven to improve project performance. 53
Question G: Multi-skilled construction personnel were used.
Having
personnel capable of performing at multiple skill levels is preferred on small projects. Small project personnel are usually expected to perform various These tasks require skills such as scheduling, estimating, Advantages of having a
project tasks.
detailing, and working with people (CII 2001b).
multi-skilled workforce includes: (1) reduced personnel dependency, (2) increased alignment and communication effectiveness, and (3) better resource utilization (CII 1998a).
Question H: In-house maintenance personnel and on-site maintenance contractors were used to support the project.
CII research (CII 2001a)
concluded that seven of nine surveyed organizations with better budget performance and five out of seven schedule performers did not maintain separate capital project and maintenance workforces. This suggests that maintaining a consistent workforce and integrating capital and maintenance work provides benefits to both schedule and budget performance.
Question I: The quality of the construction for the project met project objectives. This question surveys the overall effectiveness of the construction
itself. It refers to the level of fulfillment of the primary project objective. It too was an addition suggest by the BM&M Small Project Team.
54
4.3.5. Start-up Practice Use
Start-up and commissioning presents specific challenges for small projects, which tend to be retrofit/renovation type efforts and often are executed in manufacturing facilities with ongoing operations. The execution of small
project start-up and commissioning activities must be thoroughly planned. Three questions are used to evaluate the effectiveness of start-up planning practices.
Question A: The start-up and commissioning team (owner, contractors, and supplier representatives) was both integrated and aligned. The start-up
and commissioning team should be composed of at least the owner, contractor, and supplier representatives. Due to time and resource limitations, usually a dedicated start-up team is non-existent on smaller projects. Since small project activities are often executed in operating facilities, coordination with the operating staff or production unit is critical. The intent of this question is to measure both the composition and coordination effort of the start-up team.
Question B: Start-up and commissioning objectives were effectively communicated. The start-up and commissioning objectives typically include:
(1) install frequent communication channels, (2) increase safety awareness, (3) align start-up logic with schedule requirements, (4) maintain budget and schedule, (5) reach a defined level of success, etc (CII 1998b). Having clear objectives is a key to implementing the specific activities during the start-up and planning phase efficiently.
55
Question C: An appropriately detailed start-up and commissioning plan was effectively implemented.
The start-up and commissioning plan should
prioritize activities with consideration for contingency options and overall strategy (CII 1998b and CII 2001b). Responsibilities of key personnel must be included in consideration.
4.3.6. Organization Practice
A programmatic approach to addressing small projects is a key to their successful management for organizations that execute a number of small projects. Six questions are designed to evaluate the characteristics of the organization.
Question A: The project was managed as part of a "small projects program" as opposed to as an individual project.
An established small projects
program allows consistent and effective management of small projects under limited resources. With a program of ongoing projects, it is possible to
aggregate project results periodically for continuous improvement purposes.
Question B: A dedicated small project core team was used to execute the project.
A dedicated team of people concentrating on a program of small
projects allows for standardized processes, systems and tools; it allows for maximum productivity across the program and allows the team to streamline the approval processes (CII 2001b).
Question C: The project team was appropriately staffed with qualified personnel (e.g. multi-discipline design engineers, technical specialists and site
56
knowledgeable personnel).
Different types of small project work requires For example,
personnel with different mixes of skills and capabilities.
environmental projects require environmental specialists, specialized equipment, lawyers, and risk management specialists. Small projects in a refinery typically require pipe designers, process engineers, and welders. The technology used by the team is an important factor as well. Whether the project requires 3D CAD or rough sketches makes a difference in the make up of the small projects team.
Question D: The project team operated as a matrix organization. The
Small Project Toolkit (CII 2001b) suggests a small project team should be operated as a matrix organization. A matrix organization can fit the clients needs better due to better reconciliation of expertise. The matrix organization allows the project team to focus on the project while functional management keeps a keen eye on quality and training of personnel, allowing corporate resources to be used more efficiently.
Question E: Supplier/contractor alliances were effectively used. There
are a variety of alliances and partnering arrangements that owners can use (CII 2001b): (1) preferred engineering contractor relationships, (2) preferred supplier relationships, (3) preferred construction contractor relationships, and (4) design-build contractor relationships. One of the benefits in using a preferred contractor is that there are little mobilization and demobilization costs. The use of preferred supplier alliances reduces the risk of late deliveries.
Question F: Team building was effectively implemented by the project team.
Team building is a project-focused process that builds and develops 57
shared goals, interdependence, trust, commitment, and accountability among team members and seeks to improve team members problem-solving skills (CII 1993b). The effectiveness of team building activities was originally surveyed in the CII BM&M Large Project Questionnaire. The BM&M Small Project Team agreed that those questions have sufficient overlap with the developed small project practice use questions regarding team composition, team alignment, communications, etc. As such, no further questions concerning this practice were developed.
4.3.7. Project Processes Practice
The basic requirements for planning, budgeting, scheduling, cost control, and staffing still applies to smaller projects. It may be possible to streamline some of these functions, but not to eliminate them. Standard processes used for large projects can be adapted to small capital programs. These modifications include evaluation of project checklists and similar process standard elements. Four questions evaluate the effectiveness of process implementation.
Question A: Standardized work processes were used and adapted to fit the needs of the project.
Experience and judgment should be used to tailor
organizational project processes. Economy and efficiency should be targeted while incorporating only the appropriate or necessary elements of the work processes (CII 2001b).
58
Question B: Progress reports were issued to team members and other stakeholders appropriately.
It is imperative that an appropriate communication
plan and system are in place to keep all the stakeholders involved and informed of project-related issues and developments (CII 2001b).
Question C: A written Project Execution Plan was effectively implemented on this project. A written plan helps assure alignment of project
stakeholders and attainment of project objectives. It facilitates project execution by documenting key execution steps or requirements. It serves as a tool to facilitate coordination, and provides a common basis of work for all project participants (CII 2001b). A sample Project Execution Plan is attached at the end of the Small Project Questionnaire in Appendix A.
Question D: Quality management systems were effectively utilized on this project.
Quality Management incorporates all activities conducted to
improve the efficiency, contract compliance and cost effectiveness of design, engineering, procurement, QA/QC, construction, and start-up elements of construction projects (CII 1994a). The effectiveness of quality management is also surveyed in the practice use section of the CII BM&M Large Project Questionnaire. This question in the small project survey serves as a wrap up question to evaluate the overall effectiveness of the quality management system (program). The Small Project Team agreed that quality questions were
appropriately addressed in front-end planning (Question J), design (Question L), and construction (Question I). developed. 59 Thus, no further specific questions were
4.3.8. Project Controls Practice
There are two levels of project controls for small projects. The first is the traditional controls on individual projects including scope, budget, and schedule management. The second level of controls includes tools and strategies to manage across a program of small projects. Both are important for successful small projects. As with large projects, managers must control the scope, cost, and schedule of each project, additionally, small project program managers need other tools to manage and control program cost, cash flow, resources, and project priorities.
Question A: Systems were in place for effectively managing changes to scope, budget, or schedule. In an operating environment, the pressure to make
last-minute changes is extremely high. The processes to manage change are identical to large projects, but a concerted effort on the part of the project management team must be made to communicate the scope early to all concerned and to commit the players to avoid change unless absolutely necessary. In
addition, the project team should identify who is authorized to make changes to the project scope, and construction personnel doing the work must clearly understand and refuse to make unauthorized changes (CII 1994b and CII 2001b).
Question B: Change orders were processed in a timely manner. This
question is intended to reinforce the concept of utilizing a project control system to control and process change orders. If a system is non-existent, processing
change orders should still be a priority since schedule can be easily delayed. The impact may lead directly to reduced project performance down the road. 60
Question C: Appropriately detailed schedules were developed and used.
With small projects, one must guard against over-planning activities, except for shutdowns, where a detailed schedule is needed. Managing small projects,
especially in an operating facility, requires that the team be flexible and responsive (CII 2001b). The Small Project Toolkit recommends that small
projects are best managed with a milestone schedule.
Question D: Detailed plans were used for plant shutdowns related to the project. Unlike other project phases, when a small project is to be executed
during a plant shutdown, the planning must be extremely detailed to minimize the length of time the operating plant is out of commission. Typically, the shutdown will be planned and coordinated for months in advance and a critical path method (CPM) schedule will often cover hourly increments during the shutdown (CII 2001b). This question was adapted from the Small Project Toolkit.
Question team. E: Weekly coordination meetings were held by the project
Effective communications are critical to the small projects success due to
short execution time and the numbers of parties involved. The Small Project Toolkit (CII 2001b) recommends communicating weekly on each project with the core team members as a key to improve project team communication.
Question expenditures. F: A system was implemented to report and control project
Given the shorter duration of small projects and the higher
potential impacts of unknowns on project costs, the importance of tracking to a budget, particularly projected spending, is all the more critical (CII 2001b).
61
Question G: The project was closed promptly after the work was completed. Once the work is completed, the project should be closed promptly
to avoid the problem of overruns caused by trickle in charges for out of scope work (CII 2001b).
4.3.9. Safety, Health and Environment Practice Use
Work involved with small projects usually involves modification and maintenance of existing facilities, making risks and hazards much greater. It is imperative that a thorough scrutiny of hazards be performed. The goal is to eliminate or reduce risks. Accidents on small projects often have a crippling effect on project safety performance metrics, due to fewer total work hours on average. It is crucial that good hazard assessment occurs and that appropriate planning be done to eliminate or reduce exposure to hazards. The questions included in the safety, health and environment section are adapted from the Zero Accident Practice section in the CII BM&M Large Project Questionnaire. These questions evaluate the effectiveness of the safety program. Areas of concern include: (1) safety procedures, (2) toolbox meetings, (3) responsibility of the safety coordinator, (4) safety incentives and contractor prequalification, (5) investigation of near misses and accidents, (6) substance abuse tests, and (7) screening for alcohol and drugs. Participants receive higher scores if the areas of concern are addressed appropriately to eliminate project risks. 62
4.3.10. Automation/Integration Practice
Many information technology tools and systems are being used in project execution today for all sized projects. The benefits of using technology on
projects include: reduced costs, shorter schedules, improved quality, reduced rework, enhanced information exchange and resource utilization, better informed team members, and smaller multi-skilled teams for project execution. The questions included in the automation/integration section are adapted from the same section in the CII BM&M Large Project Questionnaire. The section is reduced, however, to include only critical work functions based on consensus from the Small Project Team. Five work functions are selected,
including: (1) detailed design, (2) procurement, (3) construction, (4) maintenance activities, and (5) project management and controls. Participants answer the level of automation for each work function based on five pre-defined categories of technology use. For project integration,
participants are ask how well each work function is integrated across all other work functions. The response is based on a 5-point Likert scale, ranging from none (no integration) to full (fully integrated).
63
Chapter 5 Descriptive Data
The Small Project Questionnaire was released for data collection commencing July 2003. Since then, respondents from owner and contractor organizations have completed 194 small project surveys and submitted data. These data are included as a part of the CII BM&M database, which has over 1,415 submitted projects collected during the 1996 to 2005 period. This chapter includes an overview of the summary statistics of the 1,415 projects in the database. Detailed breakdown are provided for the small projects. The chapter further discusses the performance metric definitions and presents charts depicting the distribution of these norms from the 194 projects. In addition, the chapter also presents the frequency distribution of the practice use questions as well as the development of the practice use indices.
5.1. DESCRIPTIVE STATISTICS
The CII BM&M program initiated data collection in 1996. Figure 3 shows a graphical summary of the types of questionnaire used, sorted by project size. Capital projects in the CII Benchmarking & Metrics program which do not fit the small project definition fall into the Large Project category; these total 964 projects. Since July 2003, there have been 194 small projects submitted that used the Small Project Questionnaire. 64 In addition, 257 small projects were
submitted using the Large Projects Questionnaire prior to the development of the Small Project Questionnaire. These projects therefore lack small project practice use data. However, they can still be used to enrich sample sizes in performance metrics analysis. The total number of small projects, as described above, is 451 and will be referred to later in this research as the Small Project Database.
1200 1000 800 600 400
194 964
Number of Projects
257
200 0 Large Project Practices Large Projects Small project practices Large Project Practices
Small Projects
Project Size Category and Questionnaire Used
Figure 3:
BM&M database by Project Size and Questionnaire Used
Table 2 shows a breakdown of organizations that participated in the data collection effort for the Small Project Database. The 451 projects included in this research come from 30 owner organizations and 27 contractor organizations. To protect company confidentially, numbers of projects submitted by each organization is suppressed. The organizations are listed in alphabetical order. 65
Table 2:
Small Project Database by Participating Organization
Contractors ABB Lummus Global Inc. ALSTOM Power Inc. Anvil Corporation BE&K Construction Company Bechtel Group, Inc. Black & Veatch BMW Constructors Inc. CDI Engineering Group, Inc. Cianbro Corporation CoSyn Technology Day & Zimmerman International, Inc. Dillingham Construction Holdings, Inc. ECI Contractor Version Flint Energy Services Ltd. Fru-Con Construction Corporation GlaxoSmithKline Alliance Graycor Jacobs Engineering Group, Inc. Johnson Controls, Inc. Kvaerner M. A. Mortenson Company Morrison Knudsen Corporation Parsons Corporation PT Rekayasa Industri Raytheon Engineers & Constructors S&B Engineers and Constructors Ltd. Watkins Engineers & Constructors
Owners 3M Abbott Laboratories Anheuser-Busch Companies, Inc. Aramco Services Company ARCO ATOFINA Bethlehem Steel Corporation Celanese Champion International Corporation CITGO Petroleum Corporation CITGO/Jacobs Alliance E.I. duPont de Nemours & Co., Inc. Eastman Chemical Company ECI Owner Eli Lilly and Company General Motors Corporation GlaxoSmithKline International Paper LTV Steel Company, Inc. NASA Naval Facilities Engineering Command NOVA Chemicals Corporation Ontario Power Generation Procter & Gamble Company Rohm and Haas Company Solutia Inc. Tennessee Valley Authority U.S. Steel United States Department of State University of Texas System
66
The newly developed small project questionnaire has drawn great interest since data collection commenced in July 2003. During the two year span, small projects consist of over 65 percent of total projects submitted to the CII BM&M General Program. A total of 18 organizations have submitted project data using the questionnaire. Twelve of 18 organizations have participated in the CII
BM&M program prior to the development of the small project questionnaire, indicating that a majority of the organizations are continued participants. Caution is warranted to prevent introducing potential bias. Equal
participation among organizations would be ideal. However, a particular owner company demonstrated heavy participation and represents about 50 percent of the newly submitted owner projects. Statistical analysis was conducted to evaluate whether this caused company bias. The Multivariate Analysis of Variance
(MANOVA) was used to test whether the mean of performance and practice use metrics of that particular company differ systematically from the remaining sample. Results included in Appendix C show insufficient evidence to conclude that company bias exists. No data were excluded as a result.
67
5.1.1.
Respondent Type
The Small Project Database consists of 259 owner projects and 192 contractor projects. The breakdown is depicted below in Figure 4. Owner
projects consist of 54.8 percent of the total while contractor data represents about 45.2 percent. Owner data are projects submitted from participating owner
organizations. Contractor data are submitted from contractor organizations and represent several different levels of project participation such as design only, engineer-procure-construct (EPC), and design-build.
Contractors 192
Owners 259
Figure 4:
5.1.2. Project Location
Small Project Database by Respondent Type
Figure 5 shows the small project database by project location. Domestic projects are the majority with a total of 396 projects, representing 87 percent of the small project data. Domestic projects refer to projects that were built in North America, excluding Mexico. Conversely, there are 55 international
projects scattered around Asia (9 projects), Europe (37 projects), the Middle East (3 projects), Mexico (2 projects), and South America (3 projects). 68
International 55
Domestic 396
Figure 5:
Small Project Database by Project Location
A detailed cross breakdown combining respondent type and project location is depicted in Figure 6. The figure shows that owners submitted a larger proportion of international projects than domestic projects.
250 217 179 150 100 42 13 0 Domestic International Owner Contractor
200
Number of Projects
50
Project Location
Figure 6:
Owner versus Contractor Participation by Project Location
69
5.1.3. Industry Group
Projects in the small project database are categorized into four types of industry groups: buildings, heavy industrial, infrastructure, and light industrial. The breakdown by project industry group is presented in Figure 7.
150 125 139
Owner Contractor
Number of Projects
100 99 75 50 25 22 0 Buildings Heavy Industrial Infrastructure Light Industrial 24 14 17 66 70
Industry Group
Figure 7:
Small Project Database by Industry Group
Heavy industrial projects make up the majority, having 99 projects on the owner side and 139 projects on the contractor side. Most heavy industrial
projects are oil refining, chemical manufacturing, pulp and paper, and electrical generating. The heavy industry sector represents more than half of the small project database (53 percent). The concentrated number of heavy industrial
projects matches the makeup of the CII membership.
70
Buildings projects rank second in the total number of projects by industry group. Building projects include laboratories, office buildings, maintenance
facilities, schools, etc. There were 62 owner building projects and 22 contractor projects. The total number of building projects (88) equates to about one-fifth of the data (19.5 percent). Light industrial projects rank third in the project count, with 87 total, including 70 from owners and 17 from contractors. The types of light industrial projects included in this research are consumer product manufacturing, automotive assembly, pharmaceutical manufacturing, and foods. Similar to
buildings, the light industrial projects also represent about one-fifth of the data (19.2 percent). Infrastructure projects are least represented with 24 projects submitted from owners and 14 from contractors. Infrastructure projects include:
water/wastewater, electrical distribution, highway projects, etc. It is interesting to note that owner participation in the building and light industrial sector is far better than contractors. Compared to the total number of projects submitted by contractors, the owners submitted three times more projects in the building sector and up to four times more in the light industrial category. The contractors have much more participation in the heavy industrial sector. Contractor heavy industrial projects actually outnumber the total amount submitted by owners, with a total of 139 versus 99, respectively.
71
5.1.4. Project Nature
Figure 8 depicts the breakdown of the small project database by project nature separated by owners and contractors. Most projects were modernizations, followed by additions, grass roots, and then maintenance. Modernization,
addition, and grass roots are capital projects. Maintenance projects can be either capital or expense projects.
175 150 151
Owner Contractor
Number of Projects
125 100 91 75 73 50 25 22 0 Addition Grass Roots Modernization Maintenance 22 13 74
5
Project Nature
Figure 8:
Small Project Database by Project Nature
The primary reason for the low number of maintenance projects is that this is a new category, added during the development of the Small Project Questionnaire. Another contributing reason is the type of work that the Most participants in the benchmarking
benchmarking participants pursue.
program are from a capital project program. Maintenance projects generally are
72
handled by a separate division or as a separate function. It is hoped that the success of the small projects benchmarking program will entice participating organizations to involve their maintenance divisions in future benchmarking activities.
5.2. PROJECT PERFORMANCE
Project performance metrics are calculated from the data surveyed in four categories: (1) cost, (2) schedule, (3) project changes, (4) and safety. The
following section will describe the definition used for metrics calculation and present the distribution of metric scores by respondent type.
5.2.1. Cost Performance Metrics
This section provides a brief overview of two major leading indicators of project cost performance: (1) Project Cost Growth and (2) Project Budget Factor. The cost performance metrics are defined such that lower numbers are more favorable.
5.2.1.1. Project Cost Growth
The Project Cost Growth metric is one of the leading indicators for measuring project cost performance. The formula for Project Cost Growth is:
73
Project Cost Growth =
Actual Total Project Cost - Initial Predicted Project Cost Initial Predicted Project Cost
Eq. 1
For owners, the actual total project cost is the total installed cost at project turnover, excluding the cost of land; and the initial predicted cost represents the budget at the time of authorization. For contractors, the actual total project cost represents the total cost to execute the final scope of work while the initial predicted project cost is the cost estimate used as the basis of contract award. Basically, the cost growth metric measures the actual (outcome) versus the predicted (planned) budget and thus indicates how well a project is planned and controlled. A zero Project Cost Growth score means that the total installed cost equals to the predicted budget. A number below zero indicates a cost savings on a project or under-run. This metric is primarily viewed as an owner metric because of contractors generally are less able to control project change orders. However, design-build contractors that conduct extensive constructability review in the planning and design phases have the capability to assist project owners improve project scope definition that could ultimately reduce change orders and lead to lower Project Cost Growth. Figure 9 indicates the distribution of Project Cost Growth by respondent type. Following procedures adopted by the CII BM&M program to preclude distortional effect on project norms, statistical outliers beyond 1.5 times the inter-quartile range were excluded from the analysis.
74
O w ner 50
C ontractor
N
Owner Mean StDev -0.07648 0.1240 234
40 Fr equency
Contractor Mean 0.08160 StDev 0.2056 N 177
30
20
10
0 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60
Figure 9:
Distribution of Project Cost Growth by Respondent Type
Owner projects have an average Project Cost Growth of -0.076 (-7.6%) while contractor projects have an average of 0.082 (8.2%). Both owner and contractor projects appear to be somewhat symmetrically distributed around the mean. However, the distributions show a peak exists around zero for both
owners and contractors. These projects are executed on budget with little or no change orders and represent in excess of 20 percent of the total sample in both cases. This indicates a real effort to achieve budgetary goals or reporting
problems, which is difficult to determine based on metric values alone. Owners and contractors have different perspectives on cost. Owners in general, are very competitive in their project cost and frequently list cost as the primary project driver. Most owners budget more money in advance to avoid funding issues caused by change orders but some owner organizations do not carry contingency in their project budget due to different project management 75
perspectives and cultures. The no contingency policy, in return, usually forces project planners to pad estimates so that projects do not suffer cost overrun. As a result, most owners tend to deliver projects under budget, causing the distribution to be slightly skewed to the left. Contractors, in general, are more affected by the variation of Project Cost Growth due to differences in perspective on project changes and the structure of the contract. Contractors typically get additional compensation for approved change orders and therefore are less motivated to drive low Project Cost Growth, causing the distribution to be slightly skewed to the right.
76
5.2.1.2. Project Budget Factor
The Project Budget Factor is a cost performance metric that explicitly accounts for change orders initiated during project execution. The definition for Project Budget Factor defined in this research is indicated below:
Actual Total Project Cost Initial Predicted Project Cost + Approved Changes
Project Budget Factor =
Eq. 2
Approved changes represent the net cost amount of all owner-authorized changes. The Project Budget Factor is a metric that provides a better indication for contractor cost performance. Unlike owners, contractors may have little or no control over project change orders. Adjusting for approved changes to the initial predicted budget lessens the cost fluctuation caused by project change orders, and provides a new baseline for cost comparison. A budget factor less than one means that the contractor performed efficiently and was able to deliver the contract for less cost. If the work is delivered at a cost higher than the planned cost plus approved changes, the budget factor is greater than one. Figure 10 depicts the distribution of Project Budget Factor by respondent type. Following procedures adopted by the CII BM&M program to preclude distortion on project norms, statistical outliers beyond 1.5 times the inter-quartile range were excluded from the analysis.
77
50
O w ner
C ontractor
Owner Mean 0.8791 StDev N 0.1134 241
40
Contractor Mean 0.9630 StDev N 0.1097 186
Fr equency
30
20
10
0 0.6 0.7 0.8 0.9 1.0 1.1 1.2 0.6 0.7 0.8 0.9 1.0 1.1 1.2
Figure 10:
Distribution of Project Budget Factor by Respondent Type
Owner projects show a near symmetric distribution and appear to be normally distributed around 0.88, on average. Project Budget Factor acts as a re-baseline cost growth metric for owners because it accounts for the additional money (approved changes usually result in increased net costs) that the owner is willing to spend in order to deliver the project. The average Project Budget Factor is low, indicating that owners are either padding estimates or employing value management practices to cut costs. The distribution for contractors shows a significant peak at 1.0, indicating most projects report that they are completed on budget. Compared to the peak of 1.0 in owner Project Cost Growth (Figure 9), the large amount of on target projects indicate that contractors are superior to owners in delivering projected budgets as planned. Contractors may actually achieve this good performance or they absorb cost overruns to maintain relationships. Perhaps they pad their 78
bids to achieve this performance, which can be unlikely due to the competitiveness of lump-sum bids. The difference is also again driven by the different perspectives on project changes.
5.2.2. Schedule Performance Metrics
Schedule performance metrics are designed to indicate whether a project is delivered on schedule, ahead of schedule, or behind. Two major metrics: (1) project schedule growth and (2) project schedule factor are presented. Schedule performance metrics are defined so that lower numbers are more favorable.
5.2.2.1. Project Schedule Growth
The formula for the calculation of project schedule growth is indicated below:
Actual Overall Project Duration - Initial Predicted Project Duration Initial Predicted Project Duration
Project Schedule Growth =
Eq. 3
The actual overall project duration starts from pre-project planning and ends at start-up, if the planning schedule is available. If planning dates are unavailable, the phase with the earliest start date is used as the overall project start date. Similarly, the phase with the latest end date is used as the overall project end date if there is no start-up data. The initial predicted project duration is the predicted duration at the time of authorization to begin design and contruction. For contractor project schedule growth, the overall project duration is not the
79
duration of the entire project but the duration needed for the contractor to complete their scope of work. Zero project schedule growth indicates that the project was delivered right on schedule. A negative number means that the project was delivered ahead of schedule and is considered more favorable. A positive number indicates that the project was delivered behind schedule. Figure 11 depicts the distribution of
project schedule growth by respondent type. Following procedures adopted by the CII BM&M program and preclude the distorting effects of project norms, statistical outliers beyond 1.5 times of the inter-quartile range were excluded from the analysis.
O w ner 60
C ontractor
Owner Mean StDev N 0.08963 0.2038 223
50
Contractor Mean 0.04213 StDev N 0.1158 149
Fr equency
40
30
20
10
0 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60 -0.30 -0.15 0.00 0.15 0.30 0.45 0.60
Figure 11:
Distribution of Project Schedule Growth by Respondent Type
80
A significant spike appears at zero Project Schedule Growth for both owner and contractor distributions, possibly due to use of standardized design or as a result of learning from prior experience. The average project schedule factor on owner projects is 0.09 and is 0.042 for contractors. When deviating from planned, more projects tend to be
delivered behind schedule. The reasons for delays perhaps include the increased complexity of working in an operating facility, late deliveries, differing site conditions, or labor shortages. Owners, in particular, seem to have a tendency to trade-off cost with schedule.
5.2.2.2. Project Schedule Factor
The project schedule factor is a schedule performance metric that accounts for the schedule impact of owner-authorized changes during project execution. Project schedule factor is defined as:
Actual Overall Project Duration Initial Predicted Project Duration + Approved Changes
Project Schedule Factor =
Eq. 4
Figure 12 indicates the distribution of project schedule factor by respondent type. Following procedures adopted by the CII BM&M program and preclude the distortional effect of project norms, statistical outliers beyond the 1.5 times of the inter-quartile range were excluded from the analysis.
81
O w ner 60
C ontractor
Owner Mean StDev N 1.009 0.1460 216
50
Contractor Mean 0.9715 StDev 0.08461 N 155
Fr equency
40
30
20
10
0 0.75 0.90 1.05 1.20 1.35 0.75 0.90 1.05 1.20 1.35
Figure 12:
Distribution of Project Schedule Factor by Respondent Type
After adjusting for the schedule impact of changes, the owner project schedule factor distributes in a more symmetric manner with a significant peak at zero project schedule factor. The average project schedule factor on owner projects is 1.009 and 0.972 for contractor projects. This result indicates that contractors may put more
emphasis on completing projects within the planned schedule.
5.2.3. Project Change Performance Metrics
Project change orders are used to calculate the change performance metric. The change cost factor metric is presented below.
5.2.3.1. Change Cost Factor
The definition for change cost factor as defined in this research is:
82
Change Cost Factor =
Total Cost of Changes Actual Total Project Cost
Eq. 5
The total cost of changes is the net cost impact of all project change orders executed during a project. A zero net cost impact would yield a change cost factor of zero, indicating that the project did not report any changes orders or that their net cost impact is zero. In some cases, the net change impact can be negative, resulting in a negative change cost factor. Figure 13 depicts the histogram of change cost factor by respondent type. Similar to the previous performance metrics, statistical outliers have been excluded.
O w ner 60
C ontractor
Owner Mean StDev N 0.05353 0.06314 213
50
Contractor Mean 0.1161 StDev 0.1265 N 151
Fr equency
40
30
20
10
0 -0.1 0.0 0.1 0.2 0.3 0.4 0.5 -0.1 0.0 0.1 0.2 0.3 0.4 0.5
Figure 13:
Distribution of Change Cost Factor by Respondent Type
A significant number of owner and contractor projects reported zero change orders. One reason may be that certain companies do not have a good 83
system in place to capture changes.
Alternatively, proper project controls,
standard design or efficiency gained from experience can lead to projects with little or no changes. Also, there could be pressure to not report changes. Contractors report more cost-increasing changes than owners. Contractors tend to track change orders more rigorously since these changes are regarded as additional work. Approved change orders may translate to additional
compensation for contractors and may be regarded as favorable and sometimes more profitable than the original contract.
5.2.4. Safety Performance Metrics
The CII BM&M Program collects safety performance data in accordance with the OSHA 300 reporting requirements as such safety performance metrics can be calculated according to OSHAs definitions. The two primary metrics reported are Total Recordable Incidence Rate (TRIR) and the Day Away, Restricted and Transfer Rate (DART).
5.2.4.1. Total Recordable Incidence Rate
The definition for the Total Recordable Incidence Rate (TRIR), as defined by OSHA and used in this research is:
84
Total Recordable Incidence Rate =
Total Number of Recordable Cases 200,000 Total Site Work Hours
Eq. 6
A recordable case (incident) is defined as a work-related illness and any injury, which results in: loss of consciousness, restriction of work or motion, transfer to another job, or requires medical treatment beyond first aid. According to the OSHA 300 reporting guideline, the total recordable incidence rate is the number of recordable incidents occurring annually among 100 full-time workers working 40 hours per week, 50 week per year (200,000 hours per job site per year). Due to a smaller average of work hours in small projects, the TRIR metric value increases dramatically with each recorded incident. The entire sample is presented in Figure 14 to show the overall distribution. A majority of the
projects did not report any incidents, which has a distorting effect on the mean value. The data shows that 156 owner projects (83 percent) and 68 contractor projects (73 percent) recorded zero recordable incidences, representing about three quarters of the distribution. Numerous scores exist in the 5 20 range, while extreme outliers extend well into the 50 60 range, far beyond normal metric values. Currently, the CII BM&M Program establishes a 3.0 IQR fence beyond the inter-quartile range as the outlier limit, stripping out only the extreme outliers. However, due the large numbers of zero values and the wide data range, the owner TRIR metric after stripping extreme outliers still contains only zero values. This result suggests that the screening effect created by stripping outliers is too strong. 85
O w ner 160 140 120 Fr equency 100 80 60 40 20 0 0 10 20 30 40 50 60 0 10 20
C ontractor
Owner Mean 2.302 StDev N Mean StDev N 7.697 188 2.517 7.066 92
Contractor
30
40
50
60
Figure 14:
Data Distribution of TRIR by Respondent Type
The TRIR metric normalizes the number of incidents to a project based on 200,000 work hours. Since the average work-hours of small projects are
significantly lower than 200,000 work hours, the TRIR metric tends to become too sensitive when incidents occur. This indicates the use of TRIR may not be a very good indicator of safety performance on small projects. A reasonable
adjustment to future calculations is to revise the 200,000 work hours standard to a lower number which would better represent the work volume on small projects. Table 3 shows the central tendency of small project work hours by respondent type. The distribution of work hours is positively skewed, as To be
indicated by the separation between the mean and median values.
conservative in constructing a work hour standard for small projects, this research suggests an upper bound of 95 percent confidence interval of the mean. A 86
preliminary number of 50,000 can be used for calculating an adjusted small project total recordable incidence rate. The adjustment would decrease the data range to 25 percent of the current distribution. Table 3: Central Tendency of Small Project Work Hours by Respondent Type
Owner
Number of Projects Mean Median 95% Confidence Interval of Mean 195 35,495 9,550 24,374 46,615
Contractor
97 41,867 15,837 25,937 57,797
All
292 37,612 12,000 28,541 46,683
5.2.4.2. Days Away, Restricted or Transfer Rate
The Days Away, Restricted or Transfer (DART) Rate is defined as:
Total Number of DART Cases 200,000 Total Site Work Hours
DART Rate =
Eq. 7
DART cases are defined as incidents that result in, days away from work, restricted activity, or transfer. The DART rate, similar to the TRIR, calculates the total number of DART cases occurring annually among 100 full-time workers working 40 hours per week, and 50 weeks per year. Figure 15 shows the DART rate metric data by respondent type. The dataset includes 185 owner projects and 90 contractor projects. For owner
projects, 169 out of 185 (91 percent) total projects had zero DART incidents. 87
The maximum rate is 41.67. For contractors, 85 out of 90 (94 percent) projects had zero DART incidents, with a maximum incidence rate of 14.17.
O w ner 180 160 140 120 100 80 60 40 20 0 0 8 16 24 32 40 0 8
C ontractor
Owner Mean StDev N 1.113 5.376 185
Contractor Mean 0.3707 StDev N 1.889 90
Fr equency
16
24
32
40
Figure 15:
Data Distribution of DART Rate by Respondent Type
Due to the smaller average number of work-hours, the current DART rate metric definition is also too sensitive to properly illustrate days away, restricted or transfer cases on small projects. The usage of the DART metric may not be a good indicator of small project safety performance. Similar to the total
recordable incidence rate, a new work hour standard can be selected to replace the current 200,000 work hour criteria with preliminary number of 50,000 for calculating an adjusted small project DART rate. The adjustment would
decrease the data range to 25 percent of the current distribution.
88
5.2.5. Phase Factors
Phase factors present invaluable information regarding the effort allocated to project phases. Two types of phase factors are calculated and presented
below. Phase cost factors show the percentage of money allocated for each project phase. Phase schedule factors indicate the percentage of time spent on each project phase.
5.2.5.1. Phase Cost Factors
Phase Cost Factors are defined as: Actual Phase Cost Actual Total Project Cost
Phase Cost Factor =
Eq. 8
The phase cost factor calculates a metric value, between zero and one, indicating the percent money spent on a particular phase compared to the total project cost (TIC). Five different phase cost factors are calculated for owners and contractors. The five phases include: (1) pre-project planning, (2) design, (3) procurement, (4) construction, and (5) start-up. The sum of all phase cost factors for a single project, typically, will be equal to one (100 percent). If a project respondent only provides partial phase cost information, phase cost factors cannot be calculated for the phases that did not report actual phase cost. Figure 16 depicts the five phase cost factors by respondent type, n values are provided under each bar to indicate the number of projects included in each breakout. The whiskers extend to the largest or smallest observation that is not classified as an outlier, which is shown as an asterisk (*) in the analysis. To
89
avoid confusion, the analysis figure did not distinguish normal outliers and extreme outliers.
1.0
1.0
0.8 Phase Cost Factor (%)
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
n=109 n=52 Owner Contractor PPP n=162 n=72 Owner Contractor Design n=120 n=58 Owner Contractor Procurement n=167 n=70 Owner Contractor Construction n=65 n=19 Owner Contractor Start-up
0.0
Figure 16:
Phase Cost Factors by Respondent Type
While owners generally participate in all phases, data are not consistently entered through all phases and missing data exists. Out of a total of 259 owner projects, the construction phase cost factor is reported the most frequently (179); and the start-up phase cost factor is reported the least (65). Table 4 shows breakpoints for the 25th percentile (first quartile), median, mean, and 75th percentile (third quartile). Only the construction phase cost factor has a mean less than its median (negative skew), all other phase cost factors have outliers at high values and indicating a positive skew. Median numbers are considered 90
more robust statistics where outliers exist. The sum of median values for the five phase factors is very close to one (1.059). The distribution of the median phase cost factors indicate a reasonable expectation of money spent on a typical owner project. Table 4: Key Statistics for Owner Phase Cost Factors
Owner
25% Pre-Project Planning Design Procurement Construction Start-up 0.018 0.055 0.101 0.446 0.010 Median 0.035 0.098 0.204 0.672 0.020 Mean 0.057 0.110 0.248 0.645 0.043 75% 0.062 0.147 0.376 0.863 0.067
Since contractors have various participation levels in the projects, it would be misleading if the data were not filtered. A design-only contractor will have a design phase cost factor of one while a construction-only contractor will have a construction phase cost factor of one. Similarly, a contractor holding an
engineering and procurement contract tends to have high phase cost factors only in design and procurement. Conversely, design and construct contractors have the opportunity to participate in more phases, providing a more meaningful allocation of project participation. Therefore, contractor data in this analysis excludes design only, construction only, and engineer and procurement (EP) only projects. Contrary to expectations; compared to owners, design and construct contractors tend to allocate more money to pre-project planning, design, procurement, and start-up, while spending less on construction. 91 Design and
construct contractors tend to be involved in multiple phases and different allocations for phase costs exist. For example, Contractor A (engineering and procurement with construction support and management) may be asked by the owner to complete mainly the planning, design, and procurement functions with limited construction support; while Contractor B (constructor) would be mainly responsible for construction work with only partial involvement in the planning and design phases. In this case, the Contractor A would likely have a higher proportion of planning, design, and procurement phase costs, and a lower proportion for construction and start-up costs. Conversely, Contractor B would have a much higher proportion of construction phase costs with lower proportions in planning, design and procurement. Table 5 shows the 25th percentile (first quartile), median, mean, and 75th percentile (third quartile) for phase cost factor by Contractor A and B. All projects in the contractor sample are manually assigned either A or B based on the project functions participated and project cost distribution characteristics. Table 5: Key Statistics for Contractor Phase Cost Factors
Contractor A (Planning & Design) 25% Median Mean 75% Pre-Project Planning Design Procurement Construction Start-up 0.076 0.113 0.266 0.218 0.021 0.117 0.156 0.377 0.359 0.046 0.124 0.162 0.420 0.319 0.069 0.174 0.226 0.526 0.431 0.124 Contractor B (Constructor) 25% Median Mean 75% 0.012 0.044 0.013 0.558 0.006 0.026 0.101 0.189 0.719 0.020 0.049 0.120 0.191 0.695 0.038 0.074 0.183 0.287 0.835 0.054
The median value may still be more representative of contractor phase cost allocation in general. For contractors involved more heavily in the planning 92
and design phases (Contractor A), their cost allocation for pre-project planning, design, and procurement would be higher while the construction and start-up phases are generally lower as compared to owners. Conversely, contractors
involved more heavily in construction (Contractor B) would indicate reverse allocation of project costs.
5.2.5.2. Phase Duration Factors
Phase Duration Factors are defined as: Actual Phase Duration Actual Overall Project Duration
Phase Duration Factor =
Eq. 9
The phase duration factor indicates the actual percentage of time allocated to each phase. Phase durations are collected in the schedule performance section of the questionnaire (Section 4.3). These durations, in the form of calendar days, are divided by the actual overall project duration, which is calculated from the project start and stop dates. Phase duration factors are calculated for each of the five phases. Figure 17 shows the array of phase duration factors by respondent type. The contractor data in the analysis excludes design only, construction only, and engineer and procurement (EP) projects. Unlike the phase cost factors, the sum of phase duration factors in a typical project will not be equal to one. The sum of phase duration factors will 93
equate to one only if phases do not overlap. Thus, fast-track projects, where schedule overlap is common, have a higher phase duration factors sum than those that are non fast-track. On the other hand, if a project has downtime between project phases, the phase duration factors sum is less than one. The data
presented in the figure indicates that each metric, with the exception of the start-up phase duration factor, demonstrates a wide range of variation. This indicates that the current dataset includes projects with significant schedule overlap and also projects with downtime between phases.
1.0
1.0
Phase Duration Factor (%)
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
n=185 n=49 Owner Contractor PPP n=151 n=68 Owner Contractor Design n=118 n=48 Owner Contractor Procurement n=161 n=60 Owner Contractor Construction n=106 n=20 Owner Contractor Start-up
0.0
Figure 17:
Phase Duration Factors by Respondent Type
Table 6 shows the quartile cutoffs for owner phase duration factors. Among all five phases, the median construction phase duration factor is longest 94
while that of start-up duration factor is shortest. All five metrics are positively skewed, indicating that extreme values tend to fall on the high (large) end. The start-up phase duration factor has the most outliers, with the largest value seen above 60 percent. The sum of all median phase duration factors is 1.256,
indicating slight schedule overlapping for most projects. Table 6: Key Statistics for Owner Phase Duration Factors
Owner
25% Pre-Project Planning Design Procurement Construction Start-up 0.139 0.174 0.124 0.256 0.007 Median 0.260 0.282 0.301 0.387 0.026 Mean 0.290 0.315 0.323 0.410 0.073 75% 0.377 0.418 0.469 0.551 0.094
Table 7 shows the key statistics for the phase duration factors by Contractor A and B. The sum of the median values of the five phase duration
factors for Contractor A is 1.663 and is 1.474 for Contractor B. Both sums of phase cost factors are greater than that of the owners (1.256), indicating that contractors tend to plan for more schedule overlaps. It is observed that
constructors spend less amount of time in pre-project planning and start-up and overlap the schedule less as compared to the planning and design contractors.
95
Table 7:
Key Statistics for Contractor Phase Duration Factors
Contractor A (Planning & Design) 25% Median Mean 75% Contractor B (Constructor) 25% Median Mean 75% 0.065 0.279 0.170 0.206 0.021 0.171 0.410 0.338 0.432 0.123 0.203 0.444 0.374 0.442 0.142 0.326 0.588 0.576 0.601 0.182
Pre-Project Planning Design Procurement Construction Start-up
0.206 0.216 0.148 0.187 0.106
0.376 0.319 0.334 0.374 0.261
0.392 0.378 0.353 0.374 0.260
0.552 0.558 0.527 0.457 0.403
It is notable that pre-project planning for a quarter of both owners and contractors was longer than one-third of the overall duration. particular, have a significant number of outliers above 0.7. Owners in These large
pre-project planning duration factors are atypical both statistically and in industry practice in general (CII 2000). categories. The first reason is that certain small projects actually do have significant lead-time and can take years to develop the final scope. These projects usually go through an initial planning effort to develop a project concept. Afterwards, project planners usually spend a minimal amount of time tracking that projects status until the project is authorized or funding is available. In this situation, the project will have a large pre-project planning phase duration factor (closer to one) while the other four phase duration factors will be fairly small (closer to zero). The other reason may be due to a reporting error resulting from a misunderstanding of definition and questionnaire instructions. Participants who misunderstood the definition of the overall project schedule were likely to enter a start and stop date that begins with project authorization and ends at project turnover. Excluding the pre-project planning duration from the overall project 96 The anomalies tend to fall into two main
schedule causes the pre-project planning phase duration factor to be calculated based on a flawed project duration. Furthermore, all other phase duration factors are inflated due to incorrect overall project duration. These cases usually
demonstrate large phase duration factors across the board, producing a sum of three or above. Although these large sums may trigger suspicion of erroneous overall project duration, the metrics may also be correct and indicate significant schedule overlap. These cases are examined during the validation stage of
project submitted to ensure that flawed reporting is reduced to a minimum level.
97
Chapter 6 Practice Use Results
Eighty-one recommended practice use questions evaluate the
implementation of small project practice use. These questions are subsequently grouped into 10 practice use areas. These areas are: (1) front-end planning, (2) design, (3) procurement, (4) construction, (5) start-up & commissioning, (6) organization, (7) project processes, (8) project controls, (9) safety, health & environment, and (10) automation / integration technology. The development of the practice use index and presentation of data follow.
6.1. PRACTICE USE INDEX DEVELOPMENT
The questions (items) in the practice use index are developed in reference to the classical test theory. According to the classical test theory (Spector 1992), the observed score (O) is a component of the true score (T) and random error (E). The true score (T), is the construct or variable of interest. The observed score is the measurement or rating obtained from an item in the practice use questions. Thus, the relationship can be expressed as: O = T + E. The
random errors should have a mean of zero and be independent of the observed score. The practice use index is developed using the summated rating scale. Based on the Summated Rating Scale Construction (Spector 1992), the summated rating scale includes four characteristics: 98
(1) The use of the scale implies that multiple items will be combined, (2) Each item must measure something that has an underlying, quantitative measurement continuum, (3) Each item response represents a rating regarding a statement, and (4) Individual items do not test knowledge or ability. The summated rating scale is based on a simple principal: With the increase of independent items, the error associated with the true score will converge such that the observed score will also converge to the true score. Eighty-one items of interest are developed for evaluation of small project practice use. These items are grouped in ten categories, namely: Front-end planning (10 items) Design (12) Procurement (6) Construction (9) Start-up & commissioning (3) Organization (6) Project processes (4) Project controls (7) Safety, health & environment (14) Automation / integration technology (10) Each item is score on a five-point scale, with ratings from (1) strongly disagree, (2) disagree, (3) neutral, (4) agree, to (5) strongly agree. The number of items in each practice is indicated in parentheses. Some questions, however, 99
are strictly defined as a yes or no question.
In this case, the intermediate
responses are not permitted, to force a selection between the strongly agree or strongly disagree rating. Not Applicable / Unknown (NA/UNK) is included as a response option. This response is designed for participants to opt-out when a practice use is not applicable or its implementation status is unknown. Although data will be lost when participants give unknown responses, the chance for reporting an inconsistent or incorrect rating is reduced.
Practice Use Index =
(Individual Item Ratings : 0 - 4) 10
No. of Valid Responses 4 Eq. 10
0 Strongly Disagree
4 Strongly Agree
1 Disagree
2 Neutral
NA/UNK X
Recommended Practices for Front-End Planning
A B C D E F G H I J Project objectives / concepts were adequately conveyed to the Front End Planning team. Front End Planning & Estimating were funded from a general program fund. The Front End Planning team was both integrated and aligned. Constructability feedback was integrated into Front End Planning. Checklists were used to ensure consistency of the Front End Planning effort. A PDRI or similar process was used to determine how well the project was defined. Contingency funds were increased as compared to large projects, ideally 3 to 5%. The Front End Planning team clearly defined the projects priorities. Front End Planning was timely and met schedule requirements. The quality of the Front End Planning met project objectives.
3 Agree
X X X X X X
0 X 3 X 3 X 3 Sum of Item Ratings 22
Practice Use Index Score = Sum of Item Ratings (22) / Number of Valid Responses (9) * 2.5 = 6.111
Figure 18:
Sample Calculation for Practice Use Index
Figure 18 gives an example of the practice use index calculation. The practice use index is calculated first by dividing the sum of item ratings by the
100
Score
3 4 2 3 1
number of valid responses within the index, and then normalizing to a scale from zero to 10. A score of zero indicates virtually no practice use while 10 Missing data and data with NA/Unk responses are not
represents full use.
considered valid responses and are excluded from the denominator. At least 50 percent of the items in a particular practice must be answered to calculate the practice use index. The number of missing data combined with NA/Unk
responses are generally below 6 percent, indicating most participants have little problem in answering the questions. It is believed that the practice use index generally results in more consistent scoring and increased reliability. This approach, called the summated rating scale is simple to use and provides good reliability and validity when item ratings are properly defined. Unreliability and inconsistency of data can be produced in several ways. Ambiguous wording of items may produce essentially random responses. If questionable terminology is used, respondents who failed to understand the intent of the question will be uncertain if the practice was implemented on the project and will tend to make random choices. Mistakes made by respondents can also produce unreliability. If the participants misread or misunderstand the question, the rating may be completely reversed. Unreliability can also be produced when a bad item, item that does not correlate with the construct, is included in the scale. A good summated rating scale must be both reliable and valid. A feature of the summated rating scale is that random error can be reduced through both the use of multiple-choice items and inclusion of more items. The following section will further discuss reliability and validity. 101
6.1.1. Reliability
Internal-consistency reliability is an indicator of how well the individual items of a scale reflect a common, underlying construct. The coefficient
(Cronbach) alpha is the statistic most often used to assess internal consistency. It is a direct function of both the number of items and their magnitude of intercorrelation. Coefficient alpha can be raised by strengthening
intercorrelation among items. However, caution should be used because even items with low intercorrelations can produce a relatively high coefficient alpha, if there are enough of them. The values of coefficient alpha are always positive, ranging from zero to one, where larger values indicate higher levels of internal consistency. Nunnally (1978) suggested that a coefficient alpha of 0.70 or above is needed to demonstrate internal consistency (Spector 1992 ). To ensure reliability in choosing items for a scale, both the item-remainder coefficient and the deleted coefficient alpha (coefficient alpha for the remaining items after deleting one item) are widely used and are available in most statistical packages. The statistics indicate the level of intercorrelation for each item. An iterative process is often used to eliminate items that demonstrate poor intercorrelation and to compare improvements of the new coefficient alpha. Table 8 summarizes standardized coefficient alpha by respondent type. The standardized coefficient alpha is based upon item covariance. Since wider range of variances are observed among certain items, the standardized coefficient alpha is used to better estimate reliability. 102
Table 8:
Standardized Coefficient Alpha Summary by Respondent Type
Practice Use Index Front-End Planning Subgroups n 156 61 95 176 73 103 149 57 92 149 57 92 169 65 104 166 68 98 180 75 105 143 60 83 78 35 43 83 13 70 Standardized Coefficient Alpha 0.848 0.919 0.745 0.875 0.921 0.808 0.773 0.852 0.713 0.773 0.852 0.713 0.881 0.874 0.896 0.707 0.553 0.721 0.761 0.852 0.632 0.800 0.838 0.807 0.957 0.987 0.791 0.918 0.953 0.913
All Contractor Owner All Design Contractor Owner All Procurement Contractor Owner All Construction Contractor Owner All Start-up Contractor Owner All Organization Contractor Owner All Project Processes Contractor Owner All Project Controls Contractor Owner All Safety Contractor Owner Automation/Integration All Contractor Owner
The levels of coefficient alpha demonstrates consistency for all ten practice use indices when all projects are used (coefficient alpha greater than 0.7). The coefficient alpha for subgroups were further calculated at the respondent level. Generally, contractor responses were more consistent than that of owners (with the exception of organization question). 103 The highest coefficient alpha
observed was 0.987 in safety practice use for contractors. Low coefficient alpha was only observed in the organization practice for contractors (0.553) and the project processes practice for owners (0.632). Appendix D provides detailed coefficient alpha analysis outputs of each practice use index including the item-remainder coefficients. No items were deleted from the index to improve the coefficient alpha. Overall, coefficient alpha analysis exhibited adequate reliability for the practice use indices.
6.1.2. Validity
The validity of a measurement relates to how well the index (scale) represents the construct of interest. However, it is often difficult to quantify the validity of the measurement alone. A more reliable approach may be to
hypothesize relationships between the construct of interest and other constructs. The hypothetical relationships may be the cause, or effect, of the construct of interest. Empirical support of the hypothesis provides evidence of validity. To support the validity of the practice use indices, it was hypothesized that a relationship exists between a projects practice use index and project performance. Higher practice use index should contribute to better project
performance and lower project risks. Acceptance of the hypothesis will provide support for practice use index validity. These analyses will be discussed in Chapter 6.
104
6.2. PRACTICE USE INDEX DATA PRESENTATION
A practice use index is calculated for each of the ten practice use sections for each project using the summated rating scale discussed previously. The 50 percent rule is applied to ensure that only individual practices with 50 percent or more valid responses are included in the practice use calculation. The
distribution and summary of key statistics of the ten practice use indices are presented below. Contractor and owner data are shown separately due to their different perspective on projects, and thus differing practice use implementation.
6.2.1. Front-End Planning
Figure 19 depicts the distribution of the front-end planning practice use index. The average value for owners is 7.094, slightly higher than the contractor average of 6.990. The contractor data also show more variation. A significant number of projects indicate full use. A contractor subgroup of low practice users exists between 4.0 and 6.0.
105
20
C ontractor
O w ner
Contractor Mean 6.990 StDev N Mean StDev N 2.216 76 7.094 1.598 110
Owner
15 Fr equency
10
5
0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
Figure 19:
Front-End Planning Practice Use Index by Respondent Type
Table 9 provides additional findings for front-end planning practice use. The skewness and kurtosis statistics are included to illustrate the shape of distribution. The skewness statistic is a measure of asymmetry. A symmetric distribution should have a skewness near zero. The kurtosis statistic indicates how the distribution is different from a normal distribution. A kurtosis near zero would indicate a normally peaked distribution, whereas a positive kurtosis typically indicates a sharper peak and negative value indicates flatter peak. Contractors demonstrate a strong negative kurtosis confirming the data is different from a normal distribution.
106
Table 9:
Front-End Planning Practice Use Statistics by Respondent Type
Front-End Planning Practice Use Statistics
Contractor Minimum Q1 Median Q3 Maximum Skewness Kurtosis 2.750 4.813 6.972 9.438 10.000 -0.040 -1.480
Owner 2.750 6.111 7.250 8.250 10.000 -0.400 -0.160
6.2.2. Design
Figure 20 illustrates the distribution of design practice use data. Both owners and contractors show a similar mean value around 7.5. Contractor index data shows more variation. A significant peak exists at 10, indicating full design
use; another peak exists at around 5.0, reflecting a lower use group.
107
25
C ontractor
O w ner
Contractor Mean 7.532 StDev N 2.047 78 7.608 1.125 110
Owner
20
Mean StDev N
Fr equency
15
10
5
0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
Figure 20:
Design Practice Use Index by Respondent Type
Table 10 provides additional results for the design practice use. More than 25 percent of contractor projects indicate full use (Q3=10.0) and peaks exist at 5.0 and 10.0. The negative kurtosis indicates the contractor data is relatively flat and different from normal distribution. Table 10: Design Practice Use Statistics by Respondent Type
Design Practice Use Statistics
Contractor Minimum Q1 Median Q3 Maximum Skewness Kurtosis 3.958 5.365 7.604 10.000 10.000 -0.180 -1.410
Owner 4.375 6.875 7.500 8.542 10.000 0.010 -0.010
108
6.2.3. Procurement
Figure 21 depicts the data distribution of the procurement practice use index. Contractors show a high negative skew with a significant number of projects indicating very high practice use. Owner data tends to distribute more evenly around the mean. Contractors show an average value of 8.602 in
procurement practice use; higher than the owners average of 7.230.
C ontractor 25
O w ner
Contractor Mean StDev N 8.602 1.633 70
20
Owner Mean 7.230 StDev N 1.681 106
Fr equency
15
10
5
0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
Figure 21:
Procurement Practice Use Index by Respondent Type
Table 11 provides additional statistics for the procurement practice use. Owners show a wider range, with a lowest value of 2.5; compared to contractors lowest value of 4.167. Half of the contractors scored 9.542 or better; a third
109
quartile (Q3) of ten indicates at least 25 percent of the contractor projects demonstrated full procurement practice use. Table 11: Procurement Practice Use Statistics by Respondent Type
Procurement Practice Use Statistics
Contractor Minimum Q1 Median Q3 Maximum Skewness Kurtosis 4.167 7.500 9.542 10.000 10.000 -1.010 -0.120
Owner 2.500 6.250 7.500 8.333 10.000 -0.560 0.370
6.2.4. Construction
Figure 22 illustrates the distribution of the construction practice use data. Owner data tends to distribute more evenly around the average of 7.570. Contractor data shows two separate peaks indicating a high use subgroup and a lower use one. The high use group scored 8.5 and above; while a low use group is observed below the mean around 6.0 to 7.0.
110
25
C ontractor
O w ner
Contractor Mean StDev N 7.849 1.408 67
20
Owner Mean 7.570 StDev 1.229 N 112
Fr equency
15
10
5
0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
Figure 22:
Construction Practice Use Index by Respondent Type
Table 12 provides additional statistics for construction practice use. Both owners and contractors show a median value of 7.5. Compared to owners,
contractor practice use has a lower kurtosis, indicating that the data differs more from a normal distribution. Table 12: Construction Practice Use Statistics by Respondent Type
Construction Practice Use Statistics
Contractor Minimum Q1 Median Q3 Maximum Skewness Kurtosis 4.444 6.875 7.500 9.167 10.000 -0.120 -1.110
Owner 5.000 6.667 7.500 8.568 10.000 0.310 -0.550
111
6.2.5. Start-up & Commissioning
Figure 23 depicts the distribution of the start-up and commissioning practice use data. Since this practice use index is only composed of three
individual items, both owner and contractor distribution indicate the existence of several peaks. Owners have a higher average practice use index than
contractors. The average practice use index for owners is 7.901, higher than the contractors average value of 7.231.
40
C ontractor
O w ner
Contractor Mean 7.231 StDev N Mean StDev N 2.543 65 7.901 1.791 104
Owner
30 Fr equency
20
10
0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
Figure 23:
Start-up & Commissioning Practice Use Index by Respondent Type
Table 13 provides additional statistics for the start-up and commissioning practice use index. Both owners and contractors have a median of 7.5.
Approximately one-quarter of owners and contractors indicate full utilization of 112
the start-up practice. A few projects reported practice use utilization of 3.5 and lower. The lowest practice index recorded for contractors and owners is 1.667 and zero, respectively. Table 13: Start-up & Commissioning Practice Use Statistics by Respondent Type
Start-up & Commissioning Practice Use Statistics
Contractor Minimum Q1 Median Q3 Maximum Skewness Kurtosis 1.667 5.000 7.500 10.000 10.000 -0.290 -1.320
Owner 0.000 6.875 7.500 10.000 10.000 -1.080 2.870
6.2.6. Organization
Figure 24 depicts the distribution of the organization practice use. Contractors exhibit more practice use with an average index value of 8.459, compared to the owners average of 6.912. The contractor data distribution
shows a significant peak of 30 projects scoring at the 8.75 9.25 bin; moreover, more than half of all contractor projects fall in or above this particular peak, indicating high practice utilization. The owners practice use index is distributed 113
across a wider range. Three owner projects have a practice score of less than 4.0.
C ontractor 30
O w ner
Contractor Mean StDev N 8.459 1.166 80
25
Owner Mean 6.912 StDev N 1.871 113
Fr equency
20
15
10
5
0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
Figure 24:
Organization Practice Use Index by Respondent Type
Table 14 provides additional statistics of the organization practice use. Contractors demonstrate better or higher practice use than owners across all statistics (minimum, first quartile, median, third quartile, and maximum). Both contractor and owner practice use data indicate a negative skew, highlighted by a stronger tilt toward higher use in the contractor distribution.
114
Table 14:
Organization Practice Use Statistics by Respondent Type
Organization Practice Use Statistics
Contractor Minimum Q1 Median Q3 Maximum Skewness Kurtosis 5.000 7.604 8.875 9.167 10.000 -0.790 -0.050
Owner 3.333 5.417 7.000 8.333 10.000 -0.130 -1.060
6.2.7. Project Processes
Figure 25 illustrates the distribution of the project processes practice use index. Several peaks exist in contractor and owner distributions, due to a
smaller number of questions (four). Contractor distribution spreads over a wider range. Twenty-three contractor projects indicate full use (10), compared to nine on the owners side.
115
25
C ontractor
O w ner
Contractor Mean 7.540 StDev N 2.202 79 7.284 1.491 110
Owner
20
Mean StDev N
Fr equency
15
10
5
0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
Figure 25:
Project Processes Practice Use Index by Respondent Type
Table 15 provides additional statistics for the project processes practice use index. The range of distribution for both contractors and owners is spread from the lowest point of 3.125 to 10. Although visual observation confirms the existence of peaks, the owners practice use distribution indicates a low kurtosis due to the concentration of two peaks in the 6.25-6.75 and the 7.25-7.75 bin.
116
Table 15:
Project Processes Practice Use Statistics by Respondent Type
Project Processes Practice Use Statistics
Contractor Minimum Q1 Median Q3 Maximum Skewness Kurtosis 3.125 5.000 8.125 10.000 10.000 -0.310 -1.420
Owner 3.125 6.250 7.500 8.281 10.000 -0.190 -0.050
6.2.8. Project Controls
Figure 26 illustrates the distribution of the project controls practice use index. Distribution of contractors and owners show similar characteristics. In both distributions, a high practice use group exists at 9.0 and above; a lower practice group centers around 7.0.
117
25
C ontractor
O w ner
Contractor Mean 7.922 StDev N 1.522 79 7.571 1.402 113
Owner
20
Mean StDev N
Fr equency
15
10
5
0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
Figure 26:
Project Controls Practice Use Index by Respondent Type
Table 16 provides additional statistics for the project controls practice use index. Table 16: Project Controls Practice Use Statistics by Respondent Type
Project Controls Practice Use Statistics
Contractor Minimum Q1 Median Q3 Maximum Skewness Kurtosis 3.750 6.786 8.333 9.286 10.000 -0.590 -0.550
Owner 3.929 6.667 7.500 8.571 10.000 -0.010 -0.460
118
6.2.9. Safety, Health & Environment
Figure 27 depicts the distribution of the safety, health and environment practice use index. Contractor scores indicate a large variance. The scores in the distribution are characterized by the existence of two subgroups, with 18 projects scoring lower than 3.25 (27 percent) and 20 projects (30 percent) indicating full practice use. Owner data appears normally distributed around the mean. The average practice use index for owners outscores the average of
contractors, 7.714 to 7.106. A comparison of mean values indicates that owners implement more safety practices.
C ontractor 30
O w ner
Contractor Mean StDev N 7.106 3.149 65
25
Owner Mean 7.714 StDev N 1.150 112
Fr equency
20
15
10
5
0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
Figure 27:
Safety, Health & Environment Practice Use Index by Respondent Type
Table 17 provides additional statistics for the safety, health and environment practice use index.
119
Table 17:
Safety, Health & Environment Practice Use Statistics by Respondent Type
Safety, Health & Environment Practice Use Statistics
Contractor Minimum Q1 Median Q3 Maximum Skewness Kurtosis 1.389 2.500 8.214 10.000 10.000 -0.730 -1.170
Owner 5.000 7.115 7.679 8.405 10.000 -0.040 -0.320
6.2.10. Automation/Integration Technology
Figure 28 illustrates the distribution of the automation / integration technology practice use index. Both owner and contractor data reveals great range and variance. Two peaks characterize the contractors distribution at These two peaks, each containing 19 projects,
6.25-6.75 and 8.75-9.25.
approximately represent 55 percent of the total contractor distribution. Owner data appears to be symmetrically distributed around the mean value of 5.2.
120
C ontractor 20
O w ner
Contractor Mean StDev N 6.477 2.030 68
Owner Mean 5.200
15 Fr equency
StDev N
2.124 107
10
5
0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
Figure 28:
Automation / Integration Technology Practice Use Index by Respondent Type
Table 18 provides additional statistics for the automation / integration technology practice use index. Contractors demonstrate higher automation /
integration technology practice use.
121
Table 18:
Automation / Integration Technology Practice Use Statistics by Respondent Type
Automation / Integration Technology Practice Use Statistics
Contractor Minimum Q1 Median Q3 Maximum Skewness Kurtosis 1.667 5.000 6.667 8.750 10.000 -0.410 -0.700
Owner 0.000 3.750 5.000 6.944 10.000 0.200 -0.340
122
Chapter 7 Data Relationships
This chapter discusses analyses exploring the relationships of the BM&M small project data. First, existing large project data are compared to the small projects in order to exemplify the difference between large and small projects. The large project dataset is used as a comparison benchmark to help broaden the perspective of the small project data. Second, data relationships within the small project set are explored, focusing on relationships between practice use and project performance. Regressions and T-tests are used to explore the
relationship between practice use index and project performance. These analysis results support both the validity of the practice use index and the research hypotheses.
7.1. DIFFERENCE BETWEEN SMALL AND LARGE PROJECTS
It is clear that there are differences in performance and practice use for small and large projects, but little research has supported this hypothesis with empirical data. The analysis below presents a quantitative comparison of
leading indicators to explore characteristics of large and small projects. The current projects within the CII BM&M Database are split into two subsets, small and large projects, for comparison. The small project dataset, presented previously, contains 451 projects. The dataset includes 194 recently submitted small projects that used the Small 123
Project Questionnaire and 257 older projects from the CII BM&M Database that used the Large Project Questionnaire. These older projects that used the Large Project Questionnaire but their cost fit the small project definition. Although this dissertation provides breakpoints for schedule duration and total work-hours, only the cost criteria was used to identify small projects in the database since it is the most widely accepted characteristic for establishing project size. Table 19 summarizes the characteristics of the small project dataset. Table 19: Characteristics of the Small Project Dataset Owner Contractor Domestic International Buildings Heavy Industrial Light Industrial Infrastructure Grass Roots Addition Modernization Maintenance
Number Percent 259 57% 192 43% 396 88% 55 12% 88 20% 238 53% 38 8% 87 19% 44 10% 147 33% 242 54% 18 4% 451 100%
Project Characteristics By respondent type
By project location By industry group
By project nature
Total number of projects
Recognizing the lack of consistency in previous definitions of small projects, this analysis took the conservative approach to the identification of large projects and thus eliminated mid-size projects (projects greater than $5 million and less than $20 million in size). The large projects selected for analysis from the CII BM&M Database are those with a total installed cost of $20 million 124
dollars or greater. By providing a sizeable separation by cost for the selection of projects, it is expected that much of the noise caused by projects close to the size boundary can be eliminated; providing a much clearer assessment of differences in small and large projects. The large project dataset consists of 573 projects with a total installed cost greater than or equal to $20 million. Table 20 provides descriptive characteristics of the large project data set. Large project data are split more evenly between owners and contractors. Similar to small projects, domestic projects represents three quarters or more of the database total. There also exist a significant number of industrial projects. Table 20: Characteristics of the Large Project Dataset Owner Contractor Domestic International Buildings Heavy Industrial Light Industrial Infrastructure Grass Roots Addition Modernization Maintenance
Number Percent 291 51% 282 49% 428 75% 145 25% 56 10% 378 66% 38 7% 101 18% 276 48% 172 30% 125 22% 0 0% 573 100%
Project Characteristics By respondent type
By project location By industry group
By project nature
Total number of projects
The metrics analyzed in this study include both project performance and practice use and are collected in the CII Benchmarking & Metrics Database. The major performance metrics include: (1) Project Cost Growth, (2) Project 125
Budget Factor, (3) Project Schedule Growth, (4) Project Schedule Factor, and (5) Change Cost Factor. As discussed in Section 4.2, both the Small and Large Project Questionnaire collect data to produce these metrics similarly by phase, but the small projects version is more brief. For the performance metrics listed above, both questionnaires collect comparable data using the same definitions. Since common practice use data only exists in the Large Project Survey dataset, comparisons can only be made for projects which originally filled out the Large Project Questionnaire. The current Large Project Questionnaire surveys 12 different practices, which were subsequently added since its initial release in 1996. Since the development of the Small Project Questionnaire, few small projects have been surveyed with the Large Project Questionnaire. Based on data availability and total sample size, five best practice indices are selected for the analysis. The selected practice use metrics include: (1) Pre-Project Planning Practice Use, (2) Constructability Practice Use, (3) Team Building Practice Use, (4) Project Change Management Practice Use, and (5) Zero Accident Technique Practice Use. Owner and contractor data are analyzed separately due to differences in project participation and perspectives. Owners are involved with the project from planning through operations and maintenance to include decommissioning, whereas contractors participate only for their scope of work. CII research has documented on numerous occasions, the differences in owner and contractor perspectives on changes. Contractors, in general, do not have control over
project changes. The treatment of changes thus impacts differences in metric 126
definitions. The Project Budget Factor and Project Schedule Factor explicitly account for the impacts of approved changes (Section 5.2). To minimize distortions in performance comparisons, outliers are eliminated using standard statistical techniques. The performance metrics used in these analyses are ratios which are unbounded, and an outlier can therefore have a disproportionate impact on norms presented. On the other hand, practice use metrics are bound from zero to 10, and are less likely to exhibit statistical outliers. As such, no outliers are removed from the practice use metrics. Analysis results are highlighted in succeeding sections with metric comparisons presented using box and whisker plots. The standard T-test
comparison including the test of equal variance is conducted to support significance. Histograms are plotted for each of the comparisons to assess the underlying assumption of normality. The two categories, Small and Large, are assumed to be independent of each other. Further assumptions regarding the population variance are made based on the significance of the test of equal variance. The pooled test is conducted when the populations assume equal variances (p-value > 0.05) and uses degrees of freedom n1+n2-2, where n1 and n2 are the sample sizes for the Small and Large categories. The Satterthwaite test uses the Satterthwaite (1946) approximation for degrees of freedom when populations assume unequal variances. Complete results and detailed statistical outputs are included in Appendix E. First, owner results are presented, followed by contractor results.
127
7.1.1. Selected Owner Comparisons
This section discusses large and small project differences using the owner data. Two performance metrics and one best practice metric were selected to illustrate the comparisons; however, a summary of all performance and practice use metrics is provided in Table 21 and Table 23. The metrics selected are (1) Project Cost Growth, (2) Change Cost Factor, and (3) Pre-Project Planning Practice Use. Section 7.1.3 summarizes the comparison results.
7.1.1.1. Project Cost Growth
Figure 29 shows the difference of owner Project Cost Growth compared by project size. The N values listed in the graph indicate the number of projects included for each project size category. Projects with missing data and statistical outliers are excluded from the sample. The final sample contains 268 large projects and 234 small projects. According the box-whisker plot, Project Cost Growth for both large and small projects appears to be normally distributed and do not violate the basic assumptions of T-test. Small project data show a greater range of cost growth compared to large projects. The standard deviation for small projects is 0.124, greater than 0.102 for large projects. 0.0025. The test of Equality of Variances indicates a p-value of
The difference is significant at the alpha level of 0.05 and the
assumption of unequal population variance is made. The small projects mean is -0.08, smaller than the mean for large projects of -0.019. This mean difference is also statistically significant at the alpha level of 0.05 (p-value<0.0001). 128
Owner Category Comparison
Global Locations with Combined Industry Groups
0.3 0.2
Project Cost Growth
0.1 0.0 -0.1 -0.2 -0.3 -0.4 -0.5 n=268 Large Projects n=234 Small Projects
Figure 29:
Owner Project Cost Growth Comparison
7.1.1.2. Change Cost Factor
Figure 30 shows the differences found for owner Change Cost Factor by project size. The sample includes 189 large projects and 213 small projects. Change Cost Factor appears to be normally distributed for both large and small projects and does not violate the basic assumptions of the T-test. Again, small project data shows a greater range with the standard deviation at 0.063, compared to 0.053 for large projects standard deviation. Although test of Equality of Variances indicates a p-value of 0.014, the difference of standard deviation between large and small projects does not imply a practical difference for real-world applications and this statistical significance might be 129
driven by the sample size. The mean value for both large and small projects is approximately 0.05 and is not statistically significant.
Owner Category Comparison
Global Locations with Combined Industry Groups
0.25 0.20
Change Cost Factor
0.15 0.10 0.05 0.00 -0.05 -0.10 n=189 Large Projects n=213 Small Projects
Figure 30:
Owner Change Cost Factor Comparison
130
7.1.1.3. Pre-Project Planning Practice Use
Figure 31 illustrates the difference of owner Pre-Project Planning Practice Use by project size. The sample includes 275 large projects and 147 small projects. Since only a portion of the BM&M small projects were submitted using the large project questionnaire containing large project best practices data, there is a drop in the available sample size in the small projects category for practice use comparisons.
Owner Category Comparison
Global Locations with Combined Industry Groups
10
Pre-Project Planning Practice Use
8 6 4 2 0 n=275 Large Projects n=147 Small Projects
Figure 31:
Owner Pre-Project Planning Practice Use Comparison
Unlike the performance metrics, the Pre-Project Planning Practice Use Index for the large and small projects appears to be skewed and has a longer tail for low scores. In this case, there exists a weakness in the assumption of
normality for the standard T-test and the results could be undermined.
131
As found in the owner performance comparisons, small project practice use data have a greater range of distribution compared to large projects. The standard deviation for small projects is 3.217, greater than the standard deviation for large projects of 2.592 with a statistically significant difference (p-value=0.0023). The mean Pre-Project Planning Practice Use for small The mean difference is also
projects is 6.171 and 7.155 for large projects.
statistically significant (p-value=0.0016), indicating that small project owners use fewer Pre-Project Planning practices.
7.1.2. Selected Contractor Comparisons
This section illustrates the contractor comparisons for large and small projects. Two performance metrics and one practice use metric are depicted; however, as with the owner data, a summary of all performance and practice use metrics is provided in Table 21 and Table 23. These analyses depicted include: (1) Project Budget Factor, (2) Change Cost Factor, and (3) Team Building Practice Use. Section 7.1.3 summarizes the comparison results.
7.1.2.1. Project Budget Factor
Figure 32 depicts the differences in Project Budget Factor between large and small project metrics for contractors. The comparison sample contains 252 large projects and 186 small projects. The box-whisker plot indicates that
Project Cost Growth for both large and small projects appears to be normally
132
distributed. The distribution of Project Budget Factor does not seem to violate normality assumption of the standard T-test. The standard deviation for the small project metric is 0.11, which is greater than the 0.078 for the large projects metric. The difference in variance is statistically significant at an alpha level of 0.05 (p-value<0.0001). The mean value for small projects is 0.963, and is 0.969 for large projects. The mean difference is not statistically significant (p-value=0.5486).
Contractor Category Comparison
Global Locations with Combined Industry Groups
1.2 1.1 1.0 0.9 0.8 0.7 n=252 Large Projects n=186 Small Projects
Figure 32:
Project Budget Factor
Contractor Project Budget Factor Comparison
7.1.2.2. Change Cost Factor
Figure 33 depicts the comparison for Change Cost Factor.
The
box-whisker plot for Change Cost Factor indicates that large projects appear more 133
normally distributed compared to small projects.
The small projects have
approximately one-fourth of projects reporting 20 percent or more Change Cost Factor which indicates a longer tail. However, the distribution does not pose a particular problem in the normality assumption for the standard T-test. Similar to the owner analysis, the small project dataset shows greater variability. The standard deviation for small projects is 0.127 and the standard deviation for large projects is 0.075. The difference in variability is statistically significant (p-value<0.001). The small projects have a mean value of 0.096, which is 57 percent greater than the large project mean of 0.055. The mean difference is proven to be statistically significant (p-value<0.0001).
Contractor Category Comparison
Global Locations with Combined Industry Groups
0.5 0.4
Change Cost Factor
0.3 0.2 0.1 0.0 -0.1 -0.2 n=184 Large Projects n=151 Small Projects
Figure 33:
Contractor Change Cost Factor Comparison
134
7.1.2.3. Team Building Practice Use
Figure 34 illustrates the difference of contractor Team Building Practice Use between large and small projects. The distribution of Team Building
Practice Use Index for small projects is skewed to the right. Similar to most practice use index distributions, the skewed distributions introduce a problem in the normality assumption. As a result, the T-test results for the practice use indices should be interpreted with caution.
Contractor Category Comparison
Global Locations with Combined Industry Groups
10
Team Building Practice Use
8 6 4 2 0 n=280 Large Projects n=105 Small Projects
Figure 34:
Contractor Team Building Practice Use Comparison
The top score for small projects is only 8.387, while the large project dataset ranges from 10 (full use) to zero (practically no use). The standard deviation of large project is 3.27, greater than the small project standard deviation of 2.547. difference This is statistically significant (p-value=0.0035) and the large projects demonstrate a higher variability in team building practice use. It 135
should be noted that higher practice use variability should be interpreted differently since it does not necessary indicate additional risks in project execution. The high variability observed in large projects indicates for owner projects a wide range of practice acceptance, which is contingent upon the companys philosophy, and management culture. It is often implemented more on the owner side since contractor would not implement such practices if the owners do not pay for the expenses. Since the Team Building Practice was originally developed for large projects, contractors executing small projects do not use the practice as much as on large projects. The mean value for the small project dataset is 2.131, nearly 50 percent lower than the large project mean of 4.667. The mean difference is statistically significant (p-value<0.0001).
136
7.1.3. Section Summary
The differences between large and small projects are summarized in this section. The differences in performance metrics are discussed first, followed by the differences in practice use metrics.
7.1.3.1. Differences in Performance Metrics by Project Size
Table 21 and Table 22 summarize the differences in performance metrics between large and small projects. The tables provide the N value, mean, and standard deviation for each performance metric by project size. presents the owner results and Table 22 the contractor results. Table 21 The tables
highlight which project size category (Large or Small) show a greater variance. The shaded cells indicate the size category with greater variance, significant at alpha=0.05 or better under the test of equal variance. Also shown are
differences in means for large and small projects. The asterisks (*) are used to indicate better performance based on the T-test results of mean differences at alpha=0.05.
137
Table 21:
Summary of Differences in Owner Performance Variation
Owner Large Projects
N mean SD
Performance Metrics Project Cost Growth
1, 2 1, 2 1, 2
Owner Small Projects
N mean SD
268 273 248 244
-0.019 0.926 0.05 1.005
0.102 0.101 0.15 0.128
234 241 223 216
-0.076 0.879 0.09 1.009 0.054
0.124 0.113 0.204 0.146 0.063
Project Budget Factor
Project Schedule Growth Project Schedule Factor
1
189 0.057 0.053 213 Change Cost Factor 1. Indicate significant difference in variance (SD), < 0.05 2. Indicate significant mean project performance difference, < 0.05
Table 22:
Summary of Differences in Contractor Performance Variation
Contractor Large Projects
N mean SD
Performance Metrics Project Cost Growth
1
Contractor Small Projects
N mean SD
269 252
1, 2
0.048 0.969 0.019 0.975
0.151 0.078 0.098 0.082
177 186 149 155
0.082 0.963 0.042 0.972 0.116
0.206 0.110 0.116 0.085 0.127
Project Budget Factor 1 Project Schedule Growth Project Schedule Factor
259 258
1, 2 184 0.066 0.075 151 Change Cost Factor 1. Indicate significant difference in variance (SD), < 0.05 2. Indicate significant mean project performance difference, < 0.05
Comparisons show that small projects generally have higher variances across the board, regardless of respondent type. The differences are generally significant, with the most significance in Project Cost Growth, Project Schedule Growth, and Change Cost Factor. The analysis results show that small project performance metrics have greater variability thus implying small project owners are exposed to higher risks in terms of project performance parameters.
138
The differences in mean values between large and small projects, suggest the following findings: Small Projects generally demonstrate higher Project Schedule Growth as compared to large projects, but have similar performance in Project Schedule Factor. Since Project Schedule Factor adjusts for the schedule impact of approved project change orders, the higher Project Schedule Growth that small projects incurred are likely a result of project changes. Contractors may track change more rigorously. Both large and
small project contractors report more changes than the owners as shown by the mean values of the Change Cost Factor. Small project contractors especially have high Change Cost Factors. Although
section 5.2.3.1 discussed that nearly 25 percent of small project contractors reported little to no changes, the high mean Change Cost Factor in Table 21 indicates that many small projects have high Change Cost Factors. The high Change Cost Factors also had a negative effect on the contractor Project Cost Growth. Owners appear to report changes in a similar manner regardless of project size. The Project Cost Growth and Project Budget Factor, however, have a significantly lower average value in small projects. As discussed previously in section 5.2.1.1 and 5.2.1.2, the low
139
average values could indicate a higher contingency reported on smaller projects.
7.1.3.2. Differences in Practice Use Metrics by Project Size
Table 23 and Table 24 summarize the differences in practice use metrics between large and small projects that were submitted using the Large Projects Survey. results. Table 23 presents the owner findings and Table 24 the contractor The tables highlight which project size category (Large or Small)
implement more best practices. The shaded cells indicate the size category with higher average practice implementation significant at alpha=0.05 or better. The tilde indicates the category with larger variance, significant at alpha=0.05 or better under the test of equal variance. Table 23: Summary of Differences in Owner Practice Use Metrics
Owner Large Projects
N mean SD
Practice Use Metrics Pre-Project Planning Constructability Team Building
1, 2 1, 2 1, 2
Owner Small Projects
N mean SD
275 238 275 237
7.155 4.694 5.036 7.940
2.592 2.632 3.100 1.691
147 138 129 142
6.171 4.162 2.107 7.422
3.217 2.558 2.547 2.328 1.961
Project Change Management
1, 2 281 7.990 1.512 138 7.187 Zero Accident Technique 1. Indicate significant mean practice use implementation difference, < 0.05 2. Indicate significant difference in variance (SD), < 0.05
140
Table 24:
Summary of Differences in Contractor Practice Use Metrics
Contractor Large Projects
N mean SD
Practice Use Metrics Pre-Project Planning Constructability Team Building
1 1, 2 1
Contractor Small Projects
N mean SD
217 209 280 196
1, 2
6.638 5.123 4.667 7.994
2.723 2.913 3.269 1.864
109 92 105 93
5.958 3.707 2.131 7.591
2.558 2.686 2.547 1.749 1.853
Project Change Management
247 8.840 1.196 51 7.736 Zero Accident Technique 1. Indicate significant mean practice use implementation difference, < 0.05 2. Indicate significant difference in variance (SD), < 0.05
Comparisons show that large projects implement the best practices better than small projects. Large projects consistently have higher average practice use scores and are statistically significant at the alpha level of 0.05, with the exception of constructability practice use for owners and project change management in contractors (both significant at 0.1). The results are consistent with past research indicating that small projects are generally less likely to implement best practices developed specifically for large projects. The differences in variance of practice use, unlike the performance metrics, should not be interpreted directly as projects risks, but to the extent that practice use correlates with performance, could imply higher risk. Figure 31, for example, compares the owner Pre-Project Planning Practice Use. The higher variance in small projects indicate that Pre-Project Planning is being implemented at both a high level and a lower level, hence the large variation. observation can be found in: Owners Project Change Management Practice Use, Owners Zero Accident Technique Practice Use, and 141 Similar
Contractors Zero Accident Technique Practice Use. Conversely, if the practice use is not implemented highly on small projects, as indicated in Figure 34 (contractor Team Building practice use), the result indicates a lower variability. This further indicates that Team Building practice is not widely accepted on contractor small projects. observations can also be found among: Contractors Pre-Project Planning Practice Use, Owners and Contractors Constructability Practice Use, Owners and Contractors Team Building Practice Use, and Contractors Project Change Management Practice Use. Similar
142
7.2. BI-VARIATE REGRESSION ANALYSIS OF PROJECT COST GROWTH AND PRACTICE USE INDEX
The bi-variate linear regression model is introduced herein to analyze the relationship between project performance and practice use. Project Cost Growth is chosen because it is the most frequently listed project driver. financial metrics have a direct impact on business decisions. Cost and
Exploring the
impact of practice implementation helps to identify the factors driving better cost performance. The summated small project practice use indices, as described in Chapter 6, are regressed individually with Project Cost Growth. These regression
analyses help identify the impact of small project practice use on Project Cost Growth. Owner and contractor projects are analyzed separately due to the
different perspectives in projects.
7.2.1. Data Preparation and Model Diagnostics
Precaution was exercised to ensure that there were no serious violations of the basic assumptions of the linear regression model. These are addressed later in Section 7.2.1.2 regarding Regression Diagnostics. The following procedures were applied to the regression models presented in this research.
143
7.2.1.1. Transformation of Practice Use Data
Small project practice use data presented in Chapter 6 demonstrates skewness and difference in the range of distribution. Standardization was used to transform the practice use data to reduce the impact of influential observations to the regression analysis. The standardization approach, in addition, offers a viable method to combine practice use indices for further analysis. Detailed discussions of analyses on combined indices are presented later in Section 7.4. The standardization approach applied to the practice use data includes two steps. First, the practice use values (X) are transformed into the standard values (Z) using the sample mean () and standard deviation (). The standardized z-values are defined as:
Z=
X-
Eq. 11
The z-values distributes in the (-, ) range as a normal distribution with a mean of zero and a standard deviation of one, if X is normally distributed. Z-values approximate the distance between the sample mean and individual values in numbers of standard deviation. For example, a z-value of one indicates the observation is one standard deviation larger than the sample mean; a z-value of negative two indicates an observation two standard deviations smaller than the sample mean. The z-values are further transformed into the cumulative distribution function (cdf) using the NORMDIST function in Microsoft Excel. The
transformed values represent the probability, or area, to the left of the standard normal curve.
144
Figure 35:
Standardization and Normal Transformation of Sample Data
Figure 35 illustrates a typical distribution of a practice use index. For example, a project score of 6.0 in the front-end planning is ultimately transformed to 0.286, indicating that the score is better than approximately 29 percent of projects (left-tail probability). The final transformed practice use values
distributes in the zero to one range, providing intuitive information regarding the position of specific observations.
7.2.1.2. Regression Diagnostics
Regression diagnostics were performed to discern if the basic underlying assumption of the regression model was violated. Analysis results were checked for normally distributed errors and constant error variance across fitted values. Four diagnostic plots were generated for each bi-variate model, including (1) the
145
normal probability plot of residuals (q-q plot), (2) the histogram of residuals, (3) the residuals versus fitted values, and (4) the residuals versus the order of the data. Appendix F contains the plots for each regression analysis. The normal probability plot and the histogram of residuals help check for normally distributed errors. The residual normal probability plot forms a straight line when errors tend to be normally distributed. All models analyzed in this research indicated that the errors are normally distributed within a reasonable tolerance. Diagnostic methods to examine the assumption of constant error variance were conducted by investigating the residual versus fitted values plot and the residuals versus the order of the data plot. constant variance within a reasonable tolerance. The errors were found to have
146
7.2.2. Owner Regression Results
Figure 36 illustrates the bi-variate relationship between owner Project Cost Growth and design practice use for small projects. Similar models were developed for all small project practices and the results are shown in Table 25.
Regression Equation: P. C. G. = -0.03819 - 0.12745 * DES INDEX R-Square: 0.0536 Pr > F: 0.0272
Figure 36:
Owner Project Cost Growth vs. Design Practice Use Scatter Plot
The model shown in Figure 36 was selected to illustrate the analysis performed. The regression slope is depicted as the solid line while the dashed
lines form the 95 percent confidence interval for individually predicted values. The regression model shows a negative slope of -0.12745, indicating that higher design practice use correlates with better (lower) Project Cost Growth. The model significance is indicated by the p-value, which is 0.0272. The low
p-value rejects the null hypothesis and supports the regression slope is different from zero. The R-square indicates the model fit, which is 0.0538 and indicates that the regression model explains 5.38 percent of the variability shown between 147
Project Cost Growth and Design Practice Use. The R-square presented is fairly low, indicating that a bi-variate model might be insufficient in explaining the relationship between the two variables. Although the bi-variate model can in some cases be used to establish a significant relationship between practice use and cost growth, a more fully specified model incorporating multiple practices would be preferable. Table 25 summarizes the bi-variate models for owner projects between Project Cost Growth and each of the practice use variables. The N values
indicate the number of samples analyzed in each pair. Statistics of the bi-variate linear regression are presented. Rankings of the relationship strength (indicated by the slope) are listed for regression coefficients. described below. Table 25: Owner Projects Practice Use Indices Regression Summary
Independent Variable Front-End Planning Design Procurement Construction Start-up & Commissioning Organization Project Processes Project Controls Safety A/I Technology N 91 91 88 93 87 94 92 94 93 91 Bi-variate Linear Regression R-Square 0.046 0.054 0.007 0.012 0.000 0.002 0.039 0.030 0.000 0.013 Slope -0.103 -0.127 -0.038 -0.046 0.007 -0.018 -0.093 -0.076 0.012 -0.049 p-value 0.046 0.027 0.439 0.291 0.900 0.676 0.061 0.096 0.862 0.281 Ranking 2 1 7 6 9 8 3 4 10 5
The key findings are
Dependent Variable Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth
Shading indicates significant results, < 0.05
148
Most of the regression models analyzed indicate negative regression slope, with the exception of start-up and safety practices. The negative slope, as
described earlier, indicates a correlation between better practice implementation and cost savings (reduction). This result overall validates H3: Better
implementation of small project practices correlates with better project performance.
Although regression slopes follow the hypothesized trend in general, only front-end planning (p-value: 0.046), design (p-value: 0.027), project processes (p-value: 0.061), and project controls (p-value: 0.096) offer statistical support indicating a significant relationship at alpha 0.1 or better. The analysis results indicate that the positive slope of start-up and safety are not significant. The R-square observed in the above models are low. This indicates that the practices failed to account for most of the variance under the bi-variate relationship. Front-end planning and design account for the highest R-square among all practice use indices, with 0.046 and 0.054, respectively. This is not unusual for this type of data and a model with only one explanatory variable. Many factors affect a projects cost growth, taken individually these variables account for only a small part of the variance in cost growth, but taken together they could explain a reasonable amount of the variance. The practices that do not show a significant relationship with Project Cost Growth may correlate better with other performance metrics such as schedule, changes, or safety. For example, safety practice use implementation generally correlates better with safety performance (CII 2003). 149
The non significant slopes should not be interpreted as the practice use implementation having no effect on Project Cost Growth. Instead, it may
indicate that the relationship may not be detectable using a simple bi-variate model with the data sample available. In addition, the low R-square values also suggests that the model may very well need to account for other factors in order to fully explain the effect of practice use implementation. Analysis between individual practice use items (questions) and Project Cost Growth can explain the impacts of each question and can ultimately lead to better performing indices as they are further developed. An item level analysis is recommended for future research in the recommendations section.
150
7.2.3. Contractor Regression Results
Table 26 summarizes the bi-variate models for contractor projects between Project Cost Growth and each of the practice use variables. Similar to owners, the N value indicates the number of projects analyzed in each pair. Key
statistics for the bi-variate regression including the R-square, slope, and p-value are listed. Rankings of the relationship strength (indicated by the slope) are listed for regression coefficients. Table 26: Contractor Projects Practice Use Indices Regression Summary
Independent Variable Front-End Planning Design Procurement Construction Start-up & Commissioning Organization Project Processes Project Controls Safety A/I Technology N 64 65 57 54 54 67 67 66 55 56 Bi-variate Linear Regression R-Square 0.234 0.293 0.004 0.138 0.267 0.002 0.182 0.207 0.371 0.002 Slope -0.257 -0.277 -0.044 -0.214 -0.288 -0.032 -0.218 -0.260 -0.320 0.034 p-value <.0001 <.0001 0.638 0.006 <.0001 0.758 0.000 0.000 <.0001 0.717 Ranking 5 3 8 7 2 9 6 4 1 10
Dependent Variable Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth Project Cost Growth
Shading indicates significant results, < 0.05
Except for the Automation/Technology Index, the regression analyses produce negative slopes indicating that increased practice use implementation on most practices leads to better cost performance. Statistical tests indicate that seven out of nine of the negative slopes from the regression analyses are significant at the alpha level of 0.05, including (1) Front-End Planning, (2) Design, (3) Construction, (4) Start-up, (5) Project Processes, (6) Project Controls, 151
and (7) Safety. Procurement and Construction practice use regressions were not significant. The R-square values indicate a considerably better fit in general compared to owner results. The safety practice use index has the largest R-square and indicates the strongest bi-variate relationship with Project Cost Growth. This however, does not warrant concluding that better Project Cost Growth can be achieved by solely improving safety practice use. The regression results very likely indicate that contractors who focus on safety practices tend to be the ones that are generally more proficient in project management practices, which ultimately lead to better project performance. Implementation of safety practices may act as a proxy for the effectiveness of contractors project management procedures in general. The non significant practice use regressions (procurement and organization) may correlate better with other areas of project performance. The relationship may be further explained by including other project factors such as project participation, alliance, and environment impacts. It may also indicate a problem in the construct of the index and that further item level analysis is needed. Compared to owner results, the regression fit for contractor data is considerably stronger in general. Further investigation was conducted to check for compositional differences in project characteristics between owners and contractors. Project characteristics investigated includes industry group, project nature, and project location. Scatter plots of the diagnostic charts are included in 152
Appendix G. The diagnostic plots did not offer clear evidence of compositional differences among these characteristics. It was observed that the distribution of individual contractor companies is a strong factor in the difference of regression outcomes between respondent types. The strong regression of contractors may be due to contributions from two companies of higher practice use projects with low Project Cost Growth and another company of lower practice use projects showing higher Project Cost Growth are highlighted in the figure below. Figure 37 illustrates the company distribution between Project Cost Growth and Design Practice Use. To protect confidentiality, the companies are identified by symbols.
Scatterplot of Project Cost Growth vs. Design Practice Use
0.6
0.4 Project Cost Growth
0.2
0.0
-0.2
-0.4 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Design Practice Use 0.7 0.8 0.9
Each symbol represents a different company
Figure 37:
Contractor Scatter Plot by Company
153
Figure 37 illustrates that two ends of the graph are dominated by two companies, labeled with the symbol () and (). The two companies also scored in a similar pattern in other practices such as front-end planning, start-up, project processes, and safety. It is observed that the variability in practice use within the two companies is low. Since the projects within the two companies are
submitted by an individual who oversees the benchmarking activity, there may be a systematic bias in the individuals response for practice implementation in different projects. However, no specific reason suggests that the data are not inaccurate. The similar practice use indices within the companies may imply that standard procedures were applied on all projects. Appendix G further gives a regression summary of a random sample that filters out 50 percent of the projects from the two companies. It is believed that the filtered sample would reduce the bias introduced by the companies. The regression summary on the filtered sample did not indicate significant deviation from findings presented earlier and given in Table 26. It is acknowledged that even after removing half of the projects submitted by the two companies, the proportion of the particular companies is still high at both ends of practice use and opportunity for bias exist. However, it was concluded that there is inconclusive evidence of a strong systematic bias based on company practice use score distribution. Thus, no data are excluded from further analysis. The high design and front-end planning scores observed on certain contractor projects indicate that it is possible, if not likely, that a long term relationship alliance exists between certain contractors and owners. 154 The
strategic alliance between owner and contractors, however, may not be representative of the entire industry group but only to a specific sector. For example, a heavy industrial owner operating a refinery plant may have alliance contractors on site to facilitate maintenance and purchase orders. In this case, contractors are likely to be familiar with the owners process and indicate high practice use scores. Owner practice use data, conversely, tends to have smaller ranges and fewer projects indicating high scores. The difference in the range of practice use data between owners and contractors may be another reason affecting the strength of correlations. Range restrictions refer to a situation where the range of the dependent variable (practice use index) is restricted, and the correlation will usually fall (Range Restrictions, Bobco 2001). This may be another possible reason for the weak correlations observed on the owner data.
7.3. T-TEST COMPARISON OF PROJECT COST GROWTH AND PRACTICE USE INDEX
To further examine the difference in Project Cost Growth values in relation to the implementation of different practice use indices, the sample of projects was divided into two categories based on the median value of individual practice use. Projects scoring equal to or better than the median value are
categorized as High practice users (implementers) and the remaining projects are categorized as Low practice users. The T-test analysis is performed to measure the differences in the Project Cost Growth mean between high and low practice users. 155
7.3.1. Owner Test Results
Table 27 summarizes the T-test comparisons of differences in Project Cost Growth mean in relation to High and Low practice users among owner projects. The pooled variance column indicates the assumptions regarding
variances made based on the test of equal variance. The pooled test assumes that the populations have equal variances (p-value > 0.05) and uses degrees of freedom n1+n2-2, where n1 and n2 are the sample sizes for the High and Low categories. The Satterthwaite values in the pooled variance column reject the null hypothesis of equal variance concluding unequal variance and uses the Satterthwaite (1946) approximation for degrees of freedom. For each category, the sample size (N), mean, and the standard deviation (SD) are displayed. The appropriate test statistic is selected based on the test of equal variance. The p-value of the appropriate test statistic is listed in the last column. The shaded cells indicate the category contributing to better Project Cost Growth, with a statistically significant result at alpha=0.05 or better. An asterisk (*) indicate the category with a smaller standard deviation (variance) which is significant at alpha=0.05 or better.
156
Table 27:
Test Variable
Owner Projects Practice Use Indices T-test Summary
Grouping Variable Group Characteristics
Category N mean SD
Test of Equal Variance (p-value) 0.256 0.008 0.321 0.607 0.239 0.075 0.477 0.813 0.977 0.325
Pooled Variance Pooled Satterthwaite Pooled Pooled Pooled Pooled Pooled Pooled Pooled Pooled
T-test (p-value) 0.114 0.029 0.390 0.048 0.895 0.977 0.181 0.006 0.777 0.591
39 -0.082 Low Project Cost Growth Front-End Planning 52 -0.124 High 38 -0.071 Low Project Cost Growth Design 1, 2 53 -0.132 High 36 -0.094 Low Project Cost Growth Procurement 52 -0.117 High 43 -0.078 Low 1 Project Cost Growth Construction 50 -0.128 High 51 -0.105 Low Start-up & Project Cost Growth Commissioning 36 -0.101 High 46 -0.109 Low Project Cost Growth Organization 48 -0.108 High 40 -0.087 Low Project Cost Growth Project Processes 52 -0.121 High 41 -0.069 Low 1 Project Cost Growth Project Controls 53 -0.139 High 46 -0.103 Low Project Cost Growth Safety 47 -0.111 High 49 -0.101 Low Project Cost Growth A/I Technology 42 -0.115 High 1. Indicate significant mean project cost growth difference, < 0.05 2. Indicate significant difference in variance (SD), < 0.05
0.137 0.116 0.145 0.097 0.109 0.128 0.115 0.124 0.134 0.111 0.142 0.109 0.129 0.116 0.123 0.119 0.126 0.126 0.111 0.129
The results of the T-test analysis can be compared to the regression summaries listed Table 25. The regression models analyze the relationship
between performance and practice use on a continuous scale, while, T-test groups high and low practice users and assess their differences. T-tests offer a clear comparison between high and low practice users and is a better method since the goal of this research does not seek to predict the outcome of practice use implementation. In addition, the test of equality of variance offers flexibility to adjust the sample variance assumption and could further detect the group with lower variability.
157
The regression summaries indicate significance in front-end planning, design, project processes, and project processes at the alpha level of 0.1 or better. At the same confidence level, design, construction, and project controls are significant under the T-test. The High practice use category consistently
correlates with better cost performance, which aligns with the negative slopes observed in the regression models. Front-end planning, construction, and project
processes differ in the significance level between methods due to a wider spread of data, indicated by a relatively low R-square. The wider spread of data causes inconsistency in grouping of practices with medium implementation efforts. It is suggested that filtering out the grey area in practice use implementation could provide more consistent results in future analyses. The largest gap of improvement opportunity, from the Low practice use category to the High practice use category, belongs to the implementation of project controls practice use. The improvement gap of project controls, design, and construction practice use is seven, six, and five percent Project Cost Growth, respectively. Since correlations exist between individual practices, the
improvement potential established using bi-variate analyses can not be summed to achieve higher savings when more than one practice use is considered. The difference in group cost growth can be best interpreted as the potential for improved performance when moving from low use to high use of the practices. The impact of combined practice indices will be discussed in Section 7.4. Front-End Planning, Procurement, and Project Processes show observed improvement in Project Cost Growth at two percent or more when comparing the 158
differences between the Low and High user category, however, these findings lack statistical significance.
7.3.2. Contractor Test Results
Table 28 summarizes the T-test comparisons of differences in Project Cost Growth mean values for High and Low practice users among contractor participants. The shaded cells indicate the category with a lower Project Cost Growth mean that is statistically significant at the alpha level of 0.05.
Table 28:
Test Variable
Contractor Projects Practice Use Indices T-test Summary
Grouping Variable Group Characteristics
Category N mean SD
Test of Equal Variance (p-value) <.0001 <.0001 0.969 0.041 <.0001 0.571 <.0001 <.0001 0.003 0.014
Pooled Variance Satterthwaite Satterthwaite Pooled Satterthwaite Satterthwaite Pooled Satterthwaite Satterthwaite Satterthwaite Satterthwaite
T-test (p-value) 0.001 0.000 0.846 0.008 0.000 0.934 0.003 0.009 0.000 0.021
32 0.128 Low 32 -0.021 High 32 0.141 Low 1, 2 Project Cost Growth Design 33 -0.029 High 29 0.065 Low Project Cost Growth Procurement 28 0.075 High 28 0.138 Low 1, 2 Project Cost Growth Construction 26 0.003 High Start-up & 29 0.158 Low Project Cost Growth 1, 2 Commissioning 25 -0.023 High 33 0.052 Low Project Cost Growth Organization 34 0.055 High 31 0.128 Low 1, 2 Project Cost Growth Project Processes 36 -0.011 High 31 0.112 Low 1, 2 Project Cost Growth Project Controls 35 -0.008 High 27 0.164 Low 1, 2 Project Cost Growth Safety 28 -0.018 High 21 0.009 Low 1, 2 Project Cost Growth A/I Technology 35 0.113 High 1. Indicate significant mean project cost growth difference, < 0.05 2. Indicate significant difference in variance (SD), < 0.05 Project Cost Growth Front-End Planning
1, 2
0.226 0.082 0.213 0.091 0.19 0.188 0.212 0.140 0.214 0.091 0.192 0.174 0.228 0.092 0.229 0.085 0.21 0.115 0.121 0.207
159
With the exception of automation/integration technology practice use, the practices that indicate statistical significance in T-test also are significant in the regression summaries shown in Table 26. These practices consistently indicate projects with high practice use implementation show a lower mean Project Cost Growth, which aligns with the negative slope shown in regression models. Compared to owners T-test results in Table 27, most Practices show significance in the mean differences. Despite smaller sample sizes, the
contractors indicate more statistical significance in practices, compared to the owners, with a total number of eight to three, respectively. The largest gap between the Low and High practice use lies in the safety category, and is 18.2 percent Project Cost Growth. Among other
practices, the improvement gap of start-up and design practices rank second and third, respectively. The improvement gap of start-up practice use is 18.1 percent and is 17 percent for design. Similar to owner results, the potential improvement of the individual practices cannot be summed; effects of combined practice use are addressed in section 7.4. The High practice use category in Front-End Planning, Design, Construction, Start-up, Project Processes, Project Controls, and Safety not only show lower Project Cost Growth but also indicate a smaller variance. The result implies that higher practice utilization on contractor projects correlates with reduced risks and better cost performance. Procurement and organization practices did not show statistically significant differences in means, which also was found for owner projects. The 160
reason that test results failed to show statistical significance are unclear and could be explored in more detail through item level analysis. item-level analysis is recommended for future research. Thus, a detailed
7.4. BI-VARIATE REGRESSION ANALYSIS OF PROJECT PERFORMANCE AND COMBINED PRACTICE USE INDEX
Bi-variate regression analysis provides useful information regarding the relationship between two variables and may provide an adequate model for preliminary analyses. A multiple regression model however, is often used to improve the bi-variate relationships. The multiple regression model explains more variance in the dependent variable compared to a bi-variate model. It also provides relative slopes and is useful in providing a more accurate estimate of the relative impact of the independent variables. However, the high correlation
among owner and contractor practice use indices, found in the following section, undermines the usage of a multiple regression analysis due to issues of multicolinearity. To facilitate further analysis of the practice use indices on project performance, various combined indices were developed. A Project Performance Index is introduced as an aggregate of project cost, schedule, and change performance. Several project practice use indices are developed by combining related small project practice use sections. Discussion detailing the development of these combined indices and their relationships follow. 161
7.4.1. Correlation among Practice Use Indices
The practice use index correlations matrices of owners and contractors are provided in Table 29 and Table 30, respectively. The Pearson correlation
coefficient is used to assess the strength and direction of linear correlation between each of the small project practices. The correlation can range from -1 to +1 indicating the direction of association. Zero indicates no linear correlation and values approaching -1 or +1 indicate strong correlation. Values in Table 29 and Table 30 indicate positive correlation for all practices as would be expected. For purposes of this analysis, only statistically significant coefficients, indicated by bold text, are considered for index development. Table 29: Owner Practice Use Index Correlation Matrix
Pearson Correlation Coefficients
Bold indicates Statistical Significance < 0.5
Front-End Planning
Project Processes
Project Controls
0.35 0.329
Organization
Construction
Procurement
Start-up
Design
Design Procurement Construction Start-up Organization Project Processes Project Controls Safety A/I Tech.
0.622 0.369 0.43 0.391 0.335 0.474 0.415 0.11 0.203 0.474 0.522 0.568 0.444 0.518 0.504 0.09 0.089 0.406 0.47 0.307 0.331 0.33 0.17 0.103
0.393 0.372 0.35 0.58 0.227 0.083
0.344 0.417 0.39 0.152 0.158
0.304 0.364 0.21 0.009
0.523 0.257 0.372
0.231
162
Safety
For both owners and contractors, the highest correlation is found between front-end planning and design. The correlation coefficients, for owner and
contractors, are 0.622 and 0.905, respectively. Strong correlation can be found between most practice use indices for both owners and contractors. In particular, projects that implement frond-end
planning and design practices generally do a good job at implementing other construction, start-up, project processes, and project controls practices as well. Overall, higher correlation coefficients are found among the contractor pairs. Table 30: Contractor Practice Use Index Correlation Matrix
Pearson Correlation Coefficients
Bold indicates Statistical Significance < 0.5
Front-End Planning
Project Processes
Project Controls
0.767 0.215
Organization
Construction
Procurement
Start-up
Design
Design Procurement Construction Start-up Organization Project Processes Project Controls Safety A/I Tech.
0.905 0.319 0.789 0.828 0.402 0.76 0.792 0.829 0.252 0.283 0.79 0.877 0.343 0.788 0.826 0.89 0.173 0.596 0.289 0.619 0.19 0.325 0.22 0.45
0.687 0.562 0.659 0.772 0.722 0.345
0.319 0.769 0.688 0.832 0.168
0.43 0.429 0.182 0.429
0.801 0.842 0.304
0.14
Notably for contractors, high correlation coefficients exist between safety practice use and other practices, with the exception of procurement, organization, and automation/integration technology. As discussed in Section 7.2.3, the
163
Safety
contractors that implement good safety practices tend to do well in other areas of practice use and have been found to have better cost performance. Conversely, on the owner side, safety practice use correlates considerably less with other practices. Projects that implement best practices may not consistently
implement the safety practices. This finding suggests that maybe a separate safety division or authority exists external to the project that oversees safety. Most owners tend to implement safety practices or have safety programs at the organization level and these other practices may not be consistently implemented on all projects due to a leaner composition of teams. In addition, owners tend to execute projects with multiple contractors, which become increasingly difficult to implement practices throughout the project, especially when there are multiple layers of communication channels between the owner and contractors. The A/I Technology index has the lowest correlation coefficients among the practice use indices. The implementation of automation or integration
technology is likely a decision made at the program (organization) level and is not considered on a project by project basis. The ambiguous responses and
correlations on A/I Technology Practice Use may also indicate that the questions are too hard to answer on small projects since the section is originally adapted from the Large Project Questionnaire.
164
7.4.2. Development of the Combined Indices
In this section, Exploratory Factor Analysis (EFA) is conducted on the ten owner practice use indices. Following the description of the factor development, owner results are discussed. Section 7.4.4 presents analysis of contractor data, using the same approach. Exploratory Factor Analysis (EFA) seeks to uncover the underlying structure of a set of variable and is often used as a variable reduction scheme. The factor analysis computes a set of eigenvalues and hence produces the uncorrelated components. This research uses both the scree test and a general guideline to account for 70 percent or more of the total variance. The scree test is a graphical method proposed by Cattell in 1966. The scree plot is depicted in Figure 38.
12 10 Eigenvalues 8 6 4 2 0 0 1 2 3 4 5 6 7 8 9 10 Rank of Components
Figure 38:
Scree plot for Practice Use Indices Factor Analysis 165
The scree test plots the magnitude of the eigenvalues on the Y axis and the components on the X axis (1st eigenvalue, 2nd eigenvalue, and so on). Generally, a breakpoint exists between components that account for a fairly large and distinct amount of variance and those components that do not. The recommendation is to retain all eigenvalues (components) before the breakpoint. In this example, the breakpoint is at three according to the scree plot, and two factors shall be retained. Since the first two factors only account for 60 percent of the total variance (generally considered low), one additional factor is retained. The three factors accounts for 68.5 percent (close to 70%) of the total variance. The factors are then rotated using the Varimax method to increase interpretability. Rotation serves to make the output more understandable and is usually necessary to facilitate the interpretation of factors. Varimax rotation is an orthogonal rotation of the factor axes to maximize the variance of the squared loadings of a factor (column) on all the variables (rows) in a factor matrix, which has the effect of differentiating the original variables by extracted factor. Each factor will tend to have either large or small loadings of any particular variable. A Varimax solution yields results which make it as easy as possible to identify each variable with a single factor and is the most common rotation option. (Stevens 2002). The rotated factors loadings are given in Table 31.
166
Table 31:
Owner Exploratory Factor Analysis Loading Matrix
Rotated Factor Pattern
Varimax Method
Factor 1 Factor 2 Factor 3
Front-End Planning Design Procurement Construction Start-up Organization Project Processes Project Controls Safety A/I Tech.
* Significant Factor Loadings
0.631* 0.747* 0.78* 0.625* 0.814* 0.28 0.482 0.495* 0.108 0.016
0.1 0.289 0.15 0.336 0.147 0.922* 0.285 0.422 0.462* 0.036
0.357 0.128 -0.024 0.058 0.103 -0.067 0.496* 0.398 0.153 0.965*
Factor loadings can be interpreted as the correlation between variables (practice use) and the factors. In this research, the significant factor loadings are defined as loadings which are 0.5 or higher. If none of the loadings in a particular practice use are greater than 0.5, the highest loading is selected. According to the rotated factor loading matrix, six practice use indices loaded on Factor 1, namely: (1) front-end planning, (2) design, (3) procurement, (4) construction, (5) start-up, and (6) project controls. This factor is best described as a general project management index for small projects as it includes the five practices that correspond to project execution plus an overarching practice in project controls. As a result, the six practice use indices were combined.
167
Composite indices were calculated by averaging the standardized value of each practice use index. The standardized approach was discussed earlier in Section 7.2.1.1. Only Factor 1 was selected to create composite index due to the fact that it accounts for the most variability in the practice use indices. Additionally, certain practice use indices loaded with Factor 2 and Factor 3 have a weak correlation between Project Cost Growth (organization, and
automation/integration technology).
Combined Practice Use Index = Average ( Standardized Front-End Planning Index, Standardized Design Index, Standardized Procurement Index, Standardized Construction Index, Standardized Start-up Index, Standardized Project Controls Index)
Eq. 12
Based on the Combined Practice Use Index, a Planning & Design Index was developed. Front-end planning and design alone were combined due to their high inter-correlation. This index is created to offer an impact assessment of preparation efforts prior to construction start. Table 32 lists the two combined practice use indices and their practice use index composition.
168
Table 32:
Front-End Planning Design Procurement Construction Start-up Organization Project Processes Project Controls Safety A/I Tech.
Composition of Practice Use Indices
Combined Practice Use Index Planning & Design Index X X X X X X X
X
Performance metrics were combined in similar manner to allow for interpretation of the relationships between project performance and practice use. The metrics included in the Project Performance Index are: (1) Project Cost Growth, (2) Project Schedule Growth, and (3) Change Cost Factor. The
standardization approach was also applied here since these metrics demonstrate different ranges of distribution. These performance metrics were chosen due to their correlations with practice use indices at the bi-variate level.
169
7.4.3. Owner Regression Results
Figure 39 depicts the scatter plot of the Project Performance Index versus the Combined Practice Use Index. The scatter plot shows a stronger slope and a slightly better fit as compared to Figure 36, which illustrates the relationship between Project Cost Growth and Design Practice Use. The use of combined indices improves the model fit of the bi-variate relationships given in Table 25.
Regression Equation: P. P. I. = -0.53947 - 0.23750 * C.P.U. INDEX R-Square: 0.0762 Pr > F: 0.0068
Figure 39:
Owner Project Performance Index vs. Combined Practice Use Index Scatter Plot
Table 33 summarizes the bi-variate linear regression model illustrating the relationship between the combined indices. The regression diagnostic plots are included in Appendix F. assumptions were found. statistical significance. No significant deviations of the basic regression All bi-variate regression models analyzed indicate
170
A better regression model is achieved when Planning & Design Index are entered as the independent variable to explain Project Cost Growth and the Project Performance Index. This model consistently achieved the highest
R-square and the strongest slope. Although the Combined Practice Use Index combines more practice use variables than the Planning & Design Index, the strength and fit of the model actually drop when the broader index is entered. It may be that Project Cost Growth does not exhibit a significant linear regression between Procurement, Construction, and Start-up Practice Uses. Including non-significant independent variables may be the main reason that the model fit failed to improve. Table 33: Owner Combined Practice Use Indices Regression Summary
Independent Variable Planning & Design Combined Practice Use Planning & Design Combined Practice Use N 89 94 90 95 Bi-variate Linear Regression R-Square 0.071 0.029 0.113 0.076 Slope -0.152 -0.110 -0.261 -0.238 p-value 0.012 0.099 0.001 0.0068 Ranking 3 4 1 2
Dependent Variable Project Cost Growth Project Cost Growth Project Performance Index Project Performance Index
Shading indicates significant results, < 0.05
The results provide support for H3: Better implementation of small
project practices correlates with better project performance.
Furthermore, the
use of combined indices in explaining the bi-variate relationships consistently increased the R-Square and slope.
171
7.4.4. Contractor Regression Results
Figure 40 depicts the scatter plot of the Project Performance Index versus the Combined Practice Use Index for contractor projects. The slope of the
bi-variate regression is strengthened using the Combined Practice Use Index as compared to results in Table 26 using single practice use indices.
Regression Equation: P. P. I. = 0.74444 - 0.38034 * C.P.U. INDEX R-Square: 0.2632 Pr > F: <.0001
Figure 40:
Contractor Project Performance Index vs. Combined Practice Use Index Scatter Plot
Table 34 provides a summary of the bi-variate linear regression model illustrating the relationship between the combined indices. The regression
diagnostic plots are included in Appendix F. No significant deviations of the basic regression assumptions were found. analyzed indicate statistical significance. All bi-variate regression models
172
Table 34:
Contractor Combined Practice Use Indices Regression Summary
Independent Variable Planning & Design Combined Practice Use Planning & Design Combined Practice Use N 64 67 69 69 Bi-variate Linear Regression R-Square 0.276 0.229 0.317 0.263 Slope -0.282 -0.320 -0.336 -0.380 p-value <0.001 <.0001 <0.001 <.0001 Ranking 4 3 2 1
Dependent Variable Project Cost Growth Project Cost Growth Project Performance Index Project Performance Index
Shading indicates significant results, < 0.05
Planning & Design regression has the highest R-square in both Project Cost Growth and Project Performance Index. Combined Practice Use Index has the strongest impact on improving both Project Cost Growth (slope: -0.320) and on performance in general, as illustrated by the Project Performance Index (slope -0.380). Contractor results strongly support the hypothesis that implementation of appropriate small project practices leads to better performance. The difference between the practice use score distribution within companies for owners and contractors, as discussed in section 7.2.3, may account for the goodness of model fit. Still other reasons can explain the stronger correlations found on the Generally, contractors are regarded as better project
contractor dataset.
implementers, have more sophisticated project management systems or procedures, and ultimately tend to report data more accurately.
173
7.5. T-TEST COMPARISON OF PROJECT PERFORMANCE AND COMBINED PRACTICE USE INDEX
T-test comparisons of the combined practice use indices offer an approach to evaluate the combined impact of practice use implementation using gap analysis. For all the analysis presented in this section, the High practice users consistently outperformed the Low practice users. The mean differences
between the High and Low practice use category, established as improvement gaps, are all statistically significant at the alpha level of 0.05.
7.5.1. Owner Test Results
Table 35 lists the T-test results for owner combined practice use indices. The Planning & Design Index has the largest improvement gap with Project Cost Growth as the test variable. The Project Cost Growth improvement from the Low to High category is 5.6 percent. statistically significant at alpha=0.05. The improvement gap for T-test results between Project Cost Growth and the Combined Practice Use Index is 4.5 percent. This difference is statistically significant at alpha=0.1. Although the Combined Practice Use Index contains more indices, its impact in improving Project Cost Growth is weaker than the Planning & Design Index. Most probably, some of the included practice use indices are less correlated with Project Cost Growth thereby weakening the composite indices. 174 The difference in the mean is
Table 35:
Test Variable
Owner Combined Practice Use Indices T-test Summary
Grouping Variable Group Characteristics
Category N mean SD
Test of Equal Variance (p-value) 0.335 0.106 0.845 0.388
Pooled Variance Pooled Pooled Pooled Pooled
T-test (p-value) 0.034 0.085 0.002 0.002
39 -0.073 Low Project Cost Growth Planning & Design 1 50 -0.129 High 42 -0.084 Low Project Cost Growth Combined Practice Use 52 -0.129 High 40 0.485 Low 1 Project Performance Index Planning & Design 50 0.378 High 43 0.476 Low Project Performance Index Combined Practice Use 1 52 0.373 High 1. Indicate significant mean project cost growth difference, < 0.05
0.13 0.112 0.14 0.11 0.155 0.16 0.148 0.168
The T-test results indicate that the High category of the composite practice use indices consistently have lower (more favorable) Project Performance Indices. The Planning & Design Index and Combined Practice Use Index were found to have similar impacts in terms of improvement gaps of -0.107 and -0.103, respectively. For example, if a project falls in the Low category of Combined Practice Use Index but improves its practice implementation into the High category, it would, on the average, out-perform approximately 10 percent of the projects in that particular data slice. The T-test of Project Cost Growth versus the Combined Practice Use Index failed to establish statistical difference of means at alpha=0.05. The test result show an observed correlation and is significant at alpha=0.1 (i.e., more practice implementation correlates with better project performance).
175
7.5.2. Contractor Test Results
Table 36 lists the T-test results of combined practice use indices for contractor projects. The test of equal variance rejected the null hypothesis on all test pairs. Thus, the T-tests were conducted under the assumption of unequal variance. The mean differences were significant at the alpha level of 0.05, and the High category consistently outperformed the Low practice use category. Table 36:
Test Variable
Contractor Combined Practice Use Indices T-test Summary
Grouping Variable Group Characteristics
Category N mean SD
Test of Equal Variance (p-value) <.0001 <.0001 0.002 0.001
Pooled Variance Satterthwaite Satterthwaite Satterthwaite Satterthwaite
T-test (p-value) 0.000 0.001 0.000 0.001
32 Low 32 High 32 Low 1, 2 Project Cost Growth Combined Practice Use 35 High 34 Low Project Performance Index Planning & Design 1, 2 32 High 34 Low 1, 2 Project Performance Index Combined Practice Use 35 High 1. Indicate significant mean project cost growth difference, < 0.05 2. Indicate significant difference in variance (SD), < 0.05
Project Cost Growth
Planning & Design 1, 2
0.136 -0.029 0.133 -0.019 0.625 0.442 0.621 0.455
0.216 0.091 0.223 0.133 0.227 0.127 0.231 0.125
Compared to owner results, the improvement gaps for contractors from Low to High practice use category is larger. The gap for Project Cost
Growth is 16 percent, for example. It should be noted that in the Low practice category, lower practice implementers generally suffer a cost overrun of 20 percent. This large Project Cost Growth is evidence that low practice
implementers usually fail in assisting owners to reduce the impact of change orders.
176
7.6. SUMMARY
Both bi-variate regression and T-test analyses indicate sufficient evidence that supports H3: Better implementation of small project practices correlates
with better project performance.
Front-End Planning and Design have the
strongest correlations among practice use indices. The regression models of both front-end planning and design indicate statistical significance for both owners and contractors at the alpha level of 0.05. In general, the analyses of practice use correlations on the contractor side show stronger correlations than for owners. This may be due to a difference in the practice use score distribution within companies as discussed in section 7.2.3. To control for this potential bias, similar analyses were conducted on a filtered sample containing only 50 percent of the suspected biased companies projects; analysis results were similar; however, and did not deviate from original findings. No specific reason suggests that the data is not accurate. Poor owner results are likely a combination of different perspectives in practice use implementation and factors other than practice use that exist in driving cost performance. Practice use indices were combined to provide a better fit and evaluate the overall effect between practice use and project performance. Two composite practice use indices were developed: (1) Planning & Design Index, and (2) Combined Practice Use Index. The improvement gap from the Low to High Combined Practice Use Index for owners and contractors is 4.5 and 15.2 percent respectively for Project Cost Growth. The strong impact of Combined Practice
177
Use Index on the contractor side suggests that high practice implementers have the ability to reduce change orders. Item level analyses may be able provide a detailed look at the specific factors (practices) which impact Project Cost Growth but is beyond the scope of this dissertation.
178
Chapter 8 Conclusions and Recommendations
This chapter completes this research by presenting research conclusions and recommendations. The research objectives are reviewed first, followed by specific conclusions on whether or not the research findings support the hypotheses. The research contributions are discussed, followed by
recommendations based on research results and future research opportunities.
8.1. REVIEW OF RESEARCH OBJECTIVES
The primary objectives of this research, as established in Section 1.3, are (1) provide a method to evaluate small project performance, (2) develop a method to evaluate the implementation of project management practices for small projects, and (3) evaluate the impact and value of suggested project management practices on small project performance. secondary objectives are: 1. 2. Review existing research on small project management. Provide a consensus definition of a small project for use in this research. 3. Summarize the differences in project performance and characteristics of large and small capital projects, based on empirical evaluation. 4. Provide project norms, i.e., statistical summaries, for project performance and practice use metrics from the data being collected for this research. 179 Based on the main objectives, the
5.
Identify key factors, i.e., project management practices, required for successful small project execution.
6.
Provide guidance to project practitioners for improved small project management.
These secondary objectives have been accomplished and support the three primary objectives. The follow section discusses the completion of objectives.
8.1.1. Review existing research on small project management
Chapter 2 provides the background of this research.
The issues and
concerns of managing small projects are included in Section 2.3. The literature indicates that small projects are best managed as a group of projects, utilize standard design processes and are controlled using milestone schedules.
8.1.2. Provide a consensus definition of a small project for use in this research
The Small Project Definition is reviewed in Section 2.2. The definition of Small Projects, as established by this research, is based not only on existing research, but also correlations drawn on empirical data analysis. The data
analysis results were critical in forming consensus regarding the quantitative breakpoints of cost, duration, and work hours. The small projects definition provides for this research should have at least one or more of the following characteristics: Total installed cost is between $100K and $5M 180
Any duration of 14 months or less Any number of site work-hours up to 100,000 The project does not require full-time project management resources or a significant percentage of company resources Any level of complexity Any type of project including maintenance and expense projects
8.1.3. Summarize the differences in project performance and characteristics of large and small capital projects
Characteristics of small and large projects are discussed in Section 7.1, followed by statistical analysis of their differences in performance and practice use. For practice use differences, the research investigated the differences of practice use metrics found in the Large Project Survey including: (1) Pre-Project Planning Practice Use, (2) Constructability Practice Use, (3) Team Building Practice Use, (4) Project Change Management Practice Use, and (5) Zero Accidents Technique Practice Use. Results indicate that small projects utilized fewer of the above practices. Comparisons also show that small projects have higher variances in performance metrics, regardless of respondent type. The analysis results imply small project owners are exposed to higher risks in terms of project performance factors.
181
8.1.4. Provide project norms, i.e., statistical summaries, for project performance and practice use metrics from the data collected for this research.
The Small Project Questionnaire was coded into a web-based, interactive, questionnaire that allows data collection continuously on the CII Benchmarking & Metrics server. This research included 194 projects submitted using the online questionnaire. Chapter 5 discusses the descriptive summaries of the small
project data and presents the definition and norms of performance metrics. Chapter 6 discusses the development of the practice use indices and presents the norms of practice use metrics.
8.1.5. Identify key factors, i.e., project management practices, required for successful small project execution.
This analysis is presented in Chapter 7. Regression and T-test analyses were conducted to explore the relationship between practice use indices and project performance. Key practices identified are front-end planning, design, and project controls. Data analysis results indicate that better implementation of these practices correlates with better project performance.
8.1.6. Provide guidance to project practitioners for improved small project management.
Overall, this research concludes that the small project practice use has a positive impact on improving project cost performance. Many organizations
pursue a portfolio of small projects in both its capital and non-capital budgets that 182
can approach half of all capital expenditures.
Practitioners should first
understand the extent of their small project portfolio in order to address its importance. In addition, they should allocate resources and address small
project as follows: Manage small projects as a program: Having a dedicated small project team allow continuity and more effective management of budget, schedule and other resources. Small project programs can be evaluated quarterly for performance checks. Increase project contingency: Smaller projects usually mean that there is less room for error in estimating a project budget. Thus, it would be ideal if the project contingency is should be increased by 2~5% as compared with larger projects. Implement Small Project Practices: This research has established a correlation between the small projects practice use and cost growth. Practitioners should fully understand the extent of the portfolio to implement relevant practices and make necessary refinements based on company culture and standard processes. Implementing key
practices in front-end planning, design, and project controls would be most beneficial and effective in early stages of a new small project program.
183
8.2. REVIEW OF RESEARCH HYPOTHESIS AND CONCLUSIONS
The research hypotheses presented in Section 1.4 are listed below for review and are followed by related conclusions based on this study:
H1:
Project
performance
and
the
implementation
of
project
management practices can be measured for small projects. Conclusion 1:
The Small Project Questionnaire was developed based on
existing literature and extensive industry feedback. The completed questionnaire was released for pilot-testing, and was subsequently coded into an interactive, online version to allow continuous data submission by the author. The data analyses conducted in this research based upon data from 194 projects submitted indicate that data are being captured with sufficient accuracy and consistency to assess both performance and practice use.
H2: There are differences in performance and practice use between large and small projects. Conclusion 2:
This research primarily investigated the difference of
project performance and practice use between large and small projects through empirical analysis. Results presented in Section 7.1 indicate that small projects implement less (1) Pre-project planning, (2) Constructability, (3) Team Building, (4) Project Change Management, and (5) Zero Accidents Technique practice use (These practices were originally designed for larger projects and may not be applicable to small projects). Results further indicated that small project 184
performance metrics show greater variability when compared to large projects. The increased variability indicates that project owners carry increased risks when managing small projects.
H3: Better implementation of small project practices correlates with better project performance. Conclusion 3:
This research quantified the correlation between small
project practices and project performance using both bi-variate regression analysis and standard T-tests. Extensive results presented in Section 7.2 through 7.6 indicate that the small project practices correlate with project performance.
8.3. RESEARCH CONTRIBUTIONS
Previous research recognized the need to develop a small project definition and suitable small project management practices. This research is an exploratory study and contributes to the body of knowledge by establishing a valid system for benchmarking small capital projects. Major contributions of this research include: 1. This research provides the literature with a more definitive definition of small projects. Based on past research and empirical analysis of
project data in the CII Benchmarking & Metrics database, and consensus of industry experts, a small project definition was developed for this research.
185
2.
This research provides the industry a means for assessing both project performance and practice use tailored for small projects. The
questionnaire developed is a flexible, interactive tool for industry participants to record and benchmark project data. The performance section of the questionnaire surveys project performance that includes cost, schedule, changes, and safety information. The practice use
section, divided into ten categories, is in a checklist format to expedite questionnaire responses. 3. This research evaluates and quantifies the differences between large and small capital projects. Differences exist in performance and practice use between large and small projects. This research first used
empirical data from the CII Benchmarking & Metrics database to quantify the differences between large and small capital projects. It was found that practices originally designed for large projects are implemented to a lesser degree by small projects, and small projects are more vulnerable to variations in cost, schedule, and change orders, and ultimately carry more risk. 4. This research provides small project norms for project performance and practice use metrics. The data collected in this research encompass 194 projects from CII organizations. Based on metrics definitions
previously established by the CII Benchmarking & Metrics Committee, project performance and practice use norms are presented by respondent type. 186
5.
This research provides the industry a validated set of project management practices to reduce risks and enhance capital effectiveness for small projects. Data correlations indicate that better
implementation of practices leads to better project performance.
8.4. FUTURE RESEARCH OPPORTUNITIES AND RECOMMENDATIONS
The research provides a comprehensive documentation on the development of a measurement tool for small projects. This research also
conducted analysis on project data to illustrate research hypotheses. Further data analysis may warrant findings that complement the results presented in this research. The following recommendations are recorded through the course of this research, and listed as follows. 1. 2. Address more suitable metrics for small project safety performance. More work should be conducted to see if the underlying constructs for practice use are effective. Detailed item level analyses can be
conducted with applications of industry knowledge and practical implications. Case studies and in-depth analysis can further evaluate the validity of practice use items (questions) that failed to indicate significant results. Finally, a revised item scoring algorithm based on the complete item level analyses can be developed to improve practice use indices.
187
3.
More analysis to assess why the practices work very well for contractors and less well for owners. Further introduce additional
factors, such as industry group, project nature, environment impacts, alliance levels, and delivery systems to regression models to better explain the correlation between practice use and project performance. 4. Further evaluate differences between large and small projects based on differences in project execution and project delivery. 5. Since high correlation exist between the small project practices, a factor analysis can be further conducted on all available items (questions) to re-categorize the practice use items to reduce redundancy in the questionnaire items. 6. Develop a small project Program questionnaire to survey the performance and standard operating procedures. Further include
dispute resolution contents in the proposed small project program questionnaire. It is suggested that the small project Program
questionnaire be surveyed quarterly and the practice use section filled out by at least three representatives. The mean of practice use
responses shall be used; in addition, the lowest and the highest response can be thrown out for more consistent results. It may be possible to integrate the small project questionnaire with the Program questionnaire to allow maximum flexibility in data collection.
188
Appendix A
The Small Project Questionnaire
189
Questionnaire Contents
1. GENERAL INFORMATION 1.1. PROJECT DESCRIPTION 1.2. PROJECT NATURE 1.3. TYPICAL PROJECT? 1.4. PROJECT DELIVERY SYSTEM 1.5. PROJECT COMPLEXITY 1.6. PROJECT SCOPE 1.7. PROJECT PARTICIPANTS (OWNERS) 1.7. PROJECT PARTICIPANTS (CONTRACTORS)
2.
PERFORMANCE/CLOSEOUT 2.1. BUDGETED AND ACTUAL PROJECT COSTS BY PHASE 2.2. PLANNED AND ACTUAL PROJECT SCHEDULE 2.3. PROJECT CHANGES 2.4. WORKHOURS AND ACCIDENT DATA 2.5. PROJECT ENVIRONMENT IMPACTS 3. PRACTICES 3.1. FRONT END PLANNING 3.2. DESIGN 3.3. PROCUREMENT 3.4. CONSTRUCTION 3.5. START-UP AND COMMISSIONING 3.6. ORGANIZATION 3.7. PROCESSES 3.8. CONTROLS 3.9. SAFETY, HEALTH AND ENVIRONMENT 3.10. AUTOMATION INTEGRATION (AI) TECHNOLOGY
190
Small Projects Questionnaire Version 1.3 General Information
Your Company Name: ______________________________________________ Project ID: _______________________________________________________ Please provide the Name that you will use to refer to this Project: ____________ Project Location: Domestic (US States or Canadian Provinces) _____________ Project Location: International (Country) ________________________________ Contact Person: (Name of knowledgeable person) _________________________ Contact's Phone: ____________________ Contact's Fax: ________________________ Contact's E-mail Address: ____________________________________________
Project Description Principle Type of Project: Choose a Project Type which best describes the project from the categories below. If the project is a mixture of two or more of those listed, select the principle type. If the project type does not appear in the list, select other under the appropriate industry group and specify the project type. Heavy Industrial Light Industrial
Chemical Manufacturing Electrical (Generating) Environmental Metals Refining/Processing Mining Natural Gas Processing Oil Exploration/Production Oil Refining Pulp and Paper Pipeline Gas Distribution Other Heavy Industrial Please specify: . Automotive Assembly Consumer Products Manufacturing Foods Microelectronics Manufacturing Office Products Manufacturing Pharmaceutical Manufacturing Pharmaceutical Labs Clean Room (Hi-Tech) Other Light Industrial Please specify: .
191
Buildings
Communications Center Dormitory/Hotel/Housing/Residential Lowrise Office (3 floors) Highrise Office (>3 floors) Hospital Laboratory Maintenance Facilities Parking Garage Physical Fitness Center Restaurant/Nightclub Retail Building School Warehouse Prison Movie Theatre Courthouse Embassy Other Buildings Please specify: . Airport
Infrastructure
Electrical Distribution Flood Control Highway Marine Facilities Navigation Rail Tunneling Water/Wastewater Telecom, Wide Area Network Other Infrastructure Please specify: .
Project Nature
Select the category that best describes the nature of this project. If your project is a combination of these natures, select the category that you would like your project to be benchmarked against. Please see the glossary for definitions.
The Project Nature was: Grass Roots Modernization Addition Maintenance Other Project Nature (Please describe)
192
Project Drivers
Select the primary driver for this project. Assume safety is a given for all projects.
The primary driver was: Cost Schedule Meeting Product Specifications Production Capacity Other (Please describe): No Primary Driver
Construction performance (cost, schedule, and safety) during project turnarounds, shutdowns, and outages may be impacted by schedule demands of the turnaround. These turnarounds may be schedule or unscheduled. Please complete the blocks below to indicate the percentage of construction work completed during turnaround. Percent construction during scheduled turnaround: Percent construction during unscheduled turnaround: Percent construction during non-turnaround:
Typical Project?
Projects submitted for benchmarking should be representative of the projects that you execute, i.e., not impacted by extraordinary factors that might influence performance or practice use metrics. If the project is not representative, it can still be submitted to be scored, however, please let us know by checking the appropriate box below. Typical Not Typical
% % %
Note: the percentages should add up to 100 %
If project is not typical, Please provide reason: a
193
Project Delivery System
Please choose the project delivery system from those listed below that most closely characterizes the delivery system used for your project. If more than one delivery system was used, select the primary system.
Delivery System Traditional Design-Bid-Build Design-Build (or EPC)
Description Serial sequence of design and construction phases; Owner contracts separately with designer and constructor. Overlapped sequence of design and construction phase; procurement normally begins during design; owner contracts with Design-Build (or EPC) contractor.
CM @ Risk
Overlapped sequence of design and construction phases; procurement normally begins during design; owner contracts separately with designer and CM @ Risk (constructor).
Multiple Design-Build
Overlapped sequence of design and construction phases; Procurement normally begins during design. Owner contracts separately with designer and multiple prime constructors.
Parallel Primes
Overlapped sequence of design and construction phases; Procurement normally begins during design. Owner contracts separately with designer and multiple prime constructors.
Other
Please describe:
Did you use a Construction Manager not @Risk in conjunction with the selected delivery system?
Yes________________ No________________
194
Project Complexity
Choose a value that best describes the level of complexity for this project as compared to other projects from all the companies within the same industry sector of similar magnitude. For example, if this is a heavy industrial project, how does it compare in complexity to other heavy industrial projects? Use the definitions below as general guidelines.
Low - Characterized by the use of proven technology, simple systems, standard designs, previously used configuration or geometry, proven construction methods, etc. High- Characterized by the use of unproven technology, complicated systems, non standard designs, new configuration or geometry, new construction methods, etc. Low Average High
1
2
3
4
5
6
7
8
9
10
Project Scope
Please provide a brief description of the project scope (what is actually being designed / constructed), limit your response to 200 words.
195
Project Participants (Owners Only) Participants: Please indicate the work percentage of each participant for each work function. Participants are categorized into Owners Personnel, Alliance Contractors, and Non-Alliance Contractors. The total work percentage should add up to 100%. If there is insufficient information for the work function, please select NA/UNK. CII Member: Was the participant that provided this phase/function a CII Member? Participants Functions
Owners Alliance Non-Alliance Personnel Contractors Contractors % % % % % % % % % % % % % % % % % %
NA/ Total Work UNK Percentage
% % % % % %
Front End Planning Detailed Design Procurement Construction Maintenance Construction Management
CII Member Functions
Owners Alliance Personnel Contractors Non-Alliance Contractors
Front End Planning Detailed Design Procurement Construction Maintenance Construction Management
Yes No
Yes No Unknown Yes No Unknown Yes No Unknown Yes No Unknown Yes No Unknown Yes No Unknown
196
Yes No Unknown Yes No Unknown Yes No Unknown Yes No Unknown Yes No Unknown Yes No Unknown
Project Participants (Contractors Only) Participants: Please estimate the work percentage of for each work function performed by your company and subcontractors. If there is insufficient information for the work function, or if you didnt perform work, please select NA/UNK. CII Member: Was the participant that provided this phase/function a CII Member?
My Company NA/ UNK Functions Front End % Planning % Detailed Design
Procurement Construction Maintenance Construction Management
CII Member
% % % %
Was the Owner of this project a CII member company? Yes Yes No No Is your company an Alliance Partner with the Owner on this project?
197
Performance/Closeout Budgeted and Actual Project Costs by Phase
Please indicate the Budgeted (Baseline) and Actual Project Costs by phase: 1. Budget amounts include contingency and correspond to funding approved at time of authorization. This is the original baseline budget, and should not be updated to include any changes since change data are collected in a later section. Metrics definitions specifically address changes as appropriate. 2. Click on the project phase links below for phase definitions and typical cost elements. 3. If this project did not include a particular phase, please select N/A. 4. The total project budget amount should include all planned expenses (excluding the cost of land) from pre-project planning through startup, including amounts estimated for in-house salaries, overhead, travel, etc. 5. The total actual project cost should include all actual project costs (excluding the cost of land) from pre-project planning through startup, including amounts expended for in-house salaries, overhead, travel, etc. 6. If you know total project costs but have incomplete phase information, you may enter as much phase information as you know and override the automatic totaling function by manually filling in the total project cost. As long as you don't click back into a phase field, your total will be accepted and recorded. 7. Enter cost in U.S. Dollars. If currency conversion is required, use the exchange rate at the midpoint of construction schedule. (Contractors Only) 8. Only enter data for your scope of work, budget amount should be the estimate at the time of contract award.
Project Phase $ Pre-Project Planning Baseline Budget (Including Contingency) $ Actual Phase Cost
NA
$
Unknown
NA
$
Unknown
Detail Design
NA
$
Unknown
NA
$
Unknown
Procurement
NA
$
Unknown
NA
$
Unknown
Construction
NA
$
Unknown
NA
$
Unknown
Startup Total Project
NA
$
Unknown
NA
$
Unknown
198
Contingency Amount within the Project
Please record the contingency amount included in total baseline budget. $______________
N/A
Unknown
Actual Total Cost of Major Equipment The purpose of this question is to determine the extent to which the overall project cost is driven by the purchase of major equipment. Please see the Equipment Reference Table provided below. Record the total purchase cost of major equipment for this project. Exclude costs for field services, bulk construction equipment (such as valves, bus duct etc.) and off-the-shelf equipment.
In the space provided below, please record the total direct cost of ENGINEERED equipment. $______________
N/A
Unknown
199
Equipment Reference Table
Examples of Major Equipment HVAC Systems Columns and Pressure Vessels Prefabricated air supply houses Towers, columns, reactors, unfired pressure vessels, bulk storage spheres, and unfired kilns; includes internals such as trays and packing. Tanks Exchangers Atmospheric storage tanks, bins, hoppers, and silos. Heat transfer equipment: tubular exchangers, condensers, evaporators, reboilers, coolers (including fin-fan coolers and coolingtowers). Direct-fired Equipment Fired heaters, furnaces, boilers, kilns, and dryers, including associated equipment such as super-heaters, air preheaters, burners, stacks, flues, draft fans and drivers, etc. Pumps Vacuum Equipment All types of liquid pumps and drivers. Mechanical vacuum pumps, ejectors, and other vacuum producing apparatus and integral auxiliary equipment. Motors Electricity Generation and Transmission 600V and above Major electrical items (e.g., unit substations, transformers, switch gear, motor-control centers, batteries, battery chargers, turbines, diesel generators). Materials-Handling Equipment Conveyers, cranes, hoists, chutes, feeders, scales and other weighing devices, packaging machines, and lift trucks. Package Units Integrated systems bought as a package (e.g., air dryers, air compressors, refrigeration systems, ion exchange systems, etc.). Special Processing Equipment Agitators, crushers, pulverizers, blenders, separators, cyclones, filters, centrifuges, mixers, dryers, extruders, fermenters, reactors, pulp and paper, and other such machinery with their drivers. Kinds of Equipment Covered
200
Planned and Actual Project Schedule
Please indicate your company's Planned Baseline and Actual Project Schedule: 1. The dates for the planned schedule should be those in effect at the start of project authorization. If you cannot provide an exact day for either the planned or actual, estimate to the nearest week. 2. The duration should be the total calendar days from start to finish of each phase. 3. Click on the project phase links below for a description of starting and stopping points for each phase. 4. If this project did not involve a particular phase please select N/A. 5. If you have incomplete phase information, you must enter overall project start and stop dates. Please enter as much phase information as possible. (Contractor Only) 6. The dates for the planned schedule should be those in effect at the estimate time of contract award.
Baseline Schedule Start mm/dd/yyyy Overall Project Start and Stop Dates Stop mm/dd/yyyy Actual Schedule Start mm/dd/yyyy Stop mm/dd/yyyy
Unknown
Unknown
Unknown
Unknown
201
Baseline DURATION CALENDAR (days) Pre-Project Planning
Actual DURATION CALENDAR (days)
NA Unknown
NA Unknown
Detail Design
NA Unknown
NA Unknown
Procurement
NA Unknown
NA Unknown
Construction
NA Unknown
NA Unknown
Startup
NA Unknown
NA Unknown
% Design Complete
What percentage of the total engineering workhours for design were completed prior to project budget authorization? What percentage of the total engineering workhours for design were completed prior to start of the construction phase? %
Unknown
%
Unknown
202
Project Changes
Please record the changes to your project by phase. Either the owner or contractor may initiate changes. 1. Changes should be included in the phase in which they were initiated. Refer to the glossary for project phase definitions to classify the changes by project phase. If you cannot provide the requested change information by phase but can provide the information for the total project, please indicate the totals. 2. Indicate whether the net impact was a decrease (-) or an increase (+) by indicating a negative number for a decrease and a positive number for an increase. If no change orders were granted during a phase, write "0" in the "Total Number" columns.
Net Schedule Project Phase Total Number of Changes Net Cost Increase (+) / Decrease (-) of Changes ($) Increase (+) / Decrease (-) of Changes (weeks) $
Detailed Design
Unknown
Unknown
$
Unknown
Construction
Totals
Unknown
Unknown
$
Unknown
203
Workhours and Accident Data
On January 31st, 2002 OSHA instituted significant changes to safety record keeping and reporting requirements. Please refer to OSHA for the new rules and definitions. A good comparison of the old and new rules may be found at: http://www.osha.gov/recordkeeping/RKside-by-side.html http://www.osha.gov/recordkeeping/RKmajorchanges.html The CII Benchmarking committee has redesigned the safety performance section so that you may report incidents accurately using the new OSHA 300 log. In the spaces below, please record the Total OSHA Number of Recordable Incidents. From that number, please break down the Number of Injuries, the Number of Illnesses and the Number of Fatalities. Also record the Total Number of OSHA DART Cases, broken out by the Number of Days Away Cases and the Number of Restricted/Transfer Cases. Next please record the number of Near Misses, the Total Site Workhours, the Percentage of Overtime Hours, and the Number of Hours in Your Normal Work Week. 1. Use the U.S. Department of Labor's OSHA definitions for recordable injuries and lost workday cases among this project's workers. If you do not track in accordance with these definitions, click Unknown in the boxes below. 2. A consolidated project OSHA 300 log is the best source for the data. and
204
Total OSHA Number Please breakdown the total Recordable Incident Cases number of Recordable (Injuries, Illnesses, Fatalities, Incident Cases by: Transfers and Restrictions) ________ Total Recordables ________ Injuries ________ Illnesses ________ Fatalities
Number of OSHA DART Cases (Days Away, Restricted or Transferred) ________ Days Away Cases ________ Restricted / Transfer Cases
Unknown
Unknown
Unknown
Near Misses Near Misses are common at many worksites. They are any work-related incident that had the potential to cause harm to an employee, damage equipment or machinery, or impact the environment in a negative way, but did not merely occur by chance. How many near misses occurred? _____________________
Do not track Unknown
Total Site Workhours ____________________
Unknown
Percentage of Overtime Hours What percentage of the workhours were overtime - above your normal work week? If the actual percentage cannot be calculated, please provide your best assessment. Answer Unknown only if you cannot make a reasonable assessment. (%)
Unknown
205
Project Environment Impacts
The following matrix is intended to assess whether environmental factors adversely or positively affected overall project performance including cost, schedule, safety, and productivity beyond the conditions for which you planned. Impacts may be assessed ranging from highly negative, to highly positive. If the factor was adequately planned for, please indicate As Planned. If it was not planned for, please indicate the impact, positive or negative. Negative impacts adversely affect the metrics and positive impacts favorably affect the metrics.
Overall Performance (Cost, Schedule, Safety, Productivity)
Hi-Neg Neg As Planned Pos Hi-Pos N/A UNK
1. Weather and Site Conditions 2. Labor Skill and Availability 3. Materials Availability 4. Project Complexity 5. Regulatory Requirements 6. Project Team Experience 7. Project Team Workload 8. Project Team Turnover 9. Coordination with Plant Shutdown
206
Practices Front End Planning
The objective of front-end planning is "to finalize, fix, and communicate project scope." For small projects, front-end planning must be completed in an environment with a compressed timeframe, few dedicated project resources, and a variable funding process. Successful projects maximize their chances of success by obtaining early and effective input from all stakeholders.
RECOMMENDED PRACTICES
Project objectives / concepts were adequately conveyed to the Front End Planning team. Front End Planning & Estimating were B funded from a general program fund or other non-project sources. The Front End Planning team (including C contractors and end users) was both integrated and aligned. Constructability feedback was integrated into D Front End Planning. Checklists were used to ensure consistency of E the Front End Planning effort. A PDRI (CII Project Definition Rating F Index) or similar process was used to determine how well the project was defined. Contingency funds were increased as G compared to large projects, ideally BY 3 to 5 PERCENTAGE POINTS. The Front End Planning team clearly defined H the projects priorities such as cost, schedule, and quality. Front End Planning was timely and met I schedule requirements. The quality of the Front End Planning met J project objectives. A
Strongly Disagree Neutral Agree Strongly NA / Disagree Agree UNK 0 1 2 3 4
207
Design The mission of the design function for performing small projects is to provide the level of design appropriate to the requirements of the project to ensure that the project is executed effectively with respect to performance, cost, and schedule. The key to completing small project design successfully includes selecting the proper design team for the given project, and making available and utilizing unique design tools specifically for the small project environment. (IF THE RECOMMENDED PRACTICE BELOW DID NOT APPLY FOR THIS PROJECT OR IF YOU DO NOT KNOW THE ANSWER, CHECK THE NA/UNK BOX)
Strongly Disagree Neutral Agree Strongly NA /
RECOMMENDED PRACTICES
A B C D E F G The scope was frozen before the start of detailed design. The design team (including site knowledgeable personnel) was both integrated and aligned. Standardized designs were incorporated into the project. Constructability feedback was integrated into the design phase. Small project checklists were used to standardize and speed up engineering. Design status review meetings were conducted as appropriate. Design changes were promptly communicated to team members. Appropriate design controls were used on the project H (e.g. budgets, schedules, checking, authorizations, scope changes). I J K L The engineering and design budget met project objectives. The engineering and design schedule met project objectives. The level of detail design on this project was adequate. (i.e. not under or over designed). The quality of engineering and design effort adequately met project objectives.
Disagree 0 1 2 3
Agree 4
UNK
208
Procurement The procurement of materials and equipment is critical to success of small projects. Due to the short duration of most small projects, errors and omissions in the procurement process have a magnified impact. It is imperative that items show up on the jobsite when they are needed, where they are needed, and without quality problems.
209
Construction
Construction of small projects in many ways is more complicated than on larger projects. The entire planning process is different. Safety risks, due to working in operating units, can often be greater. Permitting poses complications. Resources and staffing issues are different. Procurement of materials, equipment rentals, field material control, and other materials management issues also are different from large projects. Project controls are done differently as well.
210
Start-up and Commissioning
Start-up and commissioning presents specific challenges for small projects, which tend to be retrofit in nature and often are executed in live manufacturing facilities. Resources are limited and risks and hazards tend to be high. Corporate or managerial attention to small projects is low or nonexistent, but the consequences of poor project performance tend to be equal to or greater than those experienced on large projects. The execution of small project start-up and commissioning activities must be thoroughly planned and completed. Mistakes or oversights on small projects can have disastrous effects, with the potential to overshadow successes in other phases.
211
Organization
For the organizations that do more than a few small projects a year, a small projects program is the key to successful management of these projects. It allows a small number of people to consistently and effectively manage many projects at one time. An understanding of the factors that influence the effectiveness of a small projects program allows organizations to make better decisions about how to formulate or re-formulate their own. A dedicated team of people concentrating on a program of small projects allows for standardized processes, systems and tools; it allows for maximum productivity across the program and allows the team to streamline the approval processes.
212
Processes
In most large industrial facilities, small projects tend to be viewed by upper management as a "program" - i.e., it is not one small project that is of concern, it is the program of many small projects that generates upper management interest. Managing a small project is much like managing a large project. The basic requirements for planning, budgeting, scheduling, cost control, and staffing still apply. It may be possible to streamline some functions, but not to eliminate them. Standard processes used for large projects can be adapted to small capital programs. These modifications include evaluation of project checklists and similar process standard elements.
213
Controls
There are two levels of project controls for small projects. The first is the traditional controls on individual projects including scope, budget and schedule management and control. The second level of controls includes tools and strategies to manage the program of small projects. Both are important for successful small projects. As with large projects, managers must control the scope, cost and schedule of each project, but in contrast, small project program managers need additional tools to manage and control program cost, cash flow, resources and project priorities.
Strongly Disagree Neutral Agree Strongly NA /
RECOMMENDED PRACTICES
A Systems were in place for effectively managing changes to scope, budget, or schedule.
Disagree 0 1 2 3
Agree 4
UNK
B Change orders were processed in a timely manner. C D E F G Appropriately detailed schedules were developed and used. Detailed plans were used for plant shutdowns related to the project. Weekly coordination meetings were held by the project team. A system was implemented to report and control project expenditures. The project was closed promptly after the work was completed.
214
Safety, Health and Environment
Whether stated or implied, safety is always the number one priority on projects. Work involved with small projects usually involves modification and maintenance of existing facilities, making risks and hazards higher. It is imperative that thorough scrutiny of hazards be the same for small projects as for larger ones. The goal is to eliminate or reduce risks associated with a small project or small project program. If your project had no accidents or near misses, please answer the questions as you would have had there been such incidents.
Strongly Disagree Neutral Agree Strongly NA /
RECOMMENDED PRACTICES
A Site safety procedures were followed for the project. Safety was a priority topic at pre-construction and construction meetings. Pre-task planning for safety was conducted by foremen C and/or other site managers. B D Safety toolbox meetings were held daily. E The project has a safety coordinator. The time commitment of the safety coordinator was appropriate. Safety orientation training was conducted for new G contractor and subcontractor employees. F H Safety incentives were used on the project. I Safety performance was a criterion for contractor selection.
Disagree 0 1 2 3
Agree 4
UNK
J Accidents were formally investigated. K Near-misses were formally investigated. Pre-employment substance abuse tests were conducted for contractor employees. Contractor employees were randomly screened for M alcohol and drugs. L N Substance abuse tests were conducted after accidents.
215
Automation Integration (AI) Technology
Many IT tools and systems are being used in project execution today. Many more tools and systems will be available soon that will further improve and enhance project execution and small projects. The benefits of using technology on projects include: reduced costs, shorter schedules, improved quality, reduced rework, better communication, enhanced information exchange and resource utilization, better informed team members, and smaller multi-skilled teams for project execution.
Referring to the use levels below, indicate the level that the A following work functions were automated (utilized computer automated systems). Detailed Design Procurement Construction Maintenance Project Management (Including Controls)
USE LEVELS None/Minimal: Little or no utilization beyond e-mail. Some: Office equivalent software, 2D CAD for detailed design. Moderate: Standalone electronic/automated engineering discipline (3D CAD) and project services systems. Nearly Full: Some automated input/output from multiple databases with automated engineering discipline design and project services systems. Full: Fully or nearly fully automated systems dominate execution of all work functions.
None Some Moderate Nearly Full Full NA / UNK None Some Moderate Nearly Full Full NA / UNK
0
1
2
3
4
Referring to the integration levels below, indicate how well B the work functions were integrated across all other work functions. Detailed Design Procurement Construction Maintenance Project Management (Including Controls)
INTEGRATION LEVELS
0
1
2
3
4
None/Minimal: Little or no integration of electronic systems/applications. Some: Manual transfer of information via hardcopy or email. Moderate: Manual and some electronic transfer between automated systems. Nearly Full: Most systems are integrated with significant human intervention for tracking inputs/outputs.
216
Appendix B
Small Project Definition Analysis
This appendix includes the meeting discussions and the statistical analysis of CII BM&M data used to establish the Small Project Definition.
Background
The Small Projects Team (Team), consisting of a panel of eight industry experts from seven organizations, including the author of this dissertation, participated in a series of workshops to develop the small project definition. The team reviewed literature presented in Section 2.2. The Manual for Special
Project Management (CII 1991) indicates a small/special project can cost up to $100M (million). The small project toolkit (CII 2001b) identifies the cost range as between $100K and $2M. The Team agreed that the anchor point of $100M defined in Manual for Special Project Management is obsolete since it was an earlier publication and from their perspective the industry currently does not regard $100M as a small project. Through these discussions the Team adopted $5M as a consensus upper limit for small project size. The Team also agreed on a lower limit of $100K to exclude insignificant projects. Three main regression models were conducted based on CII BM&M data to identify correlations between total project cost, project duration, and total craft work-hours. Due to the skewness in project cost, duration, and craft work-hours,
217
the variables were transformed using the natural log function (ln) to meet the basic regression assumptions. The scatter-plot, along with regression models and R-square are shown in Figure B-1, Figure B-2, and Figure B-3. All models are significant at alpha = 0.05. Figure B-1, regression between project cost and craft work-hours, has the best fit among the three models developed. The regression model indicated an R-square of 0.6985, suggesting an excellent fit of the data.
20
19
17
Ln(Craft Work-Hour, hrs)
16
14
13
Correlation = 0.836 y = 1.0021x - 4.6772 2 R = 0.6985
11
10
8
7
5 10 11 12 13 14 15 16 17 18 19 20 21 22
Ln(Total Project Cost, $)
Figure B-1: Total Project Cost versus Craft Work-Hour Regression
218
6.4
5.6
Ln(Total Actual Duration, wks)
4.8
4.0
Correlation = 0.472
3.2
y = 0.1838x + 1.3422 2 R = 0.2228
2.4
1.6
0.8 7.5 8.5 9.5 10.5 11.5 12.5 13.5 14.5 15.5 16.5 17.5 18.5 19.5 20.5 21.5 22.5
Ln(Total Project Cost, $)
Figure B-2: Total Project Cost versus Actual Project Duration
20.0
17.5
Ln(Craft Work-Hour, hrs)
15.0
12.5
10.0
Correlation = 0.471 y = 1.4119x + 6.0428 2 R = 0.2216
7.5
5.0 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5
Ln(Project Actual Duration, wks)
Figure B-3: Actual Project Duration versus Craft Work-Hours
219
Using Regressions to Predict Anchor Points
Referring to Figure B-1, three project cost anchor points: $2M, $5M, and $10M were evaluated as control variables to predict the craft work hours based on the regression model. These cost anchor points were suggested by the Team since they have been established in their organizations, as well as mentioned in literature. Table B-1 lists the prediction outcome of craft work-hours. Table B-1: Prediction of Total Craft Work-hours based on Project Cost Regression
(1) Project Cost $2,000,000 $5,000,000 $10,000,000 (2) Ln Project Cost Ln (1) 14.51 15.42 16.12 (3) Ln Craft Work-Hours 1.0021*(2)-4.6772 9.86 10.78 11.47 (4) Predicted Craft Work-hours Exp(3) 19,186 48,057 96,254
A similar approach was used to predict the (project) actual duration using the regression model in Figure B-2. Results are summarized in Table B-2. Table B-2: Prediction of Actual Duration based on Project Cost Regression
(1) Project Cost (2) Ln Project Cost Ln (1) 14.51 15.42 16.12 (3) Ln Actual Duration 0.1838*(2)+1.3442 4.01 4.18 4.30 (4) Predicted Actual Duration (weeks) Exp(3) 55.09 65.19 74.05 (5) Predicted Actual Duration (months) (4)/4.3 12.81 15.16 17.22
$2,000,000 $5,000,000 $10,000,000
220
Based on the regression model established between actual duration and craft work-hour, it is still possible to derive a set of predicted craft work-hours based on the predicted actual durations (column 3) in Table B-2. Table B-3 shows the predicted craft work-hours based on the regression presented in Figure B-3. These values differ significantly from Table B-1 and are likely overestimated through error compounded by combining predictions of the previous models. Table B-3: Prediction of Actual Duration based on Project Cost Regression
(1) Predicted Actual Duration (weeks) 55.09 65.19 74.05 (2) Ln Project Cost Ln (1) 4.01 4.18 4.30 (3) Ln Craft Work-Hours 1.4119*(2)+6.0428 11.70 11.94 12.12 (4) Predicted Craft Work-hours Exp(3) 120,928 153,390 183,618
The path of analysis is illustrated in Figure B-4. It should be noted that since all three regression models do not fully explain the bi-variate relationship (R-square not equal to 1). The predicted craft work-hours will not be equal due to error in residuals.
221
Figure B-1
Table B-1 Craft Work Hours (Direct Path)
Project Cost
($2M, $5M, $10M)
Figure B-2 Figure B-3
Table B-2 Actual Duration
Table B-3 Craft Work Hours (Indirect Path)
Figure B-4: Comparison of Craft Work-Hours Prediction
After analyzing the relationships among project cost, craft work hours, and project duration provided in Tables B-1, B-2, and B-3. There was concern with the prediction accuracy of the models caused by skew in the data. This concern and consideration of previous research findings led to additional analyses of duration and work-hour data. The Team was comfortable with establishing an upper limit of $5M for the small project definition, but felt that skew in duration and work hour data had to be addressed. The following sections illustrate the
methodology for establishing upper definition limits for work hours and duration.
222
Anchoring the Craft Work-Hour
Figure B-5 plots the cumulative percent of craft work hours for project less than $5M. Based on the figure, the Team decided that an upper bound of 100,000 work hours would be a suitable limit since it covers 85% of all projects less than $5M and eliminates most of the skew in craft work hours. The 100,000 point was more consistent with the expert opinions and also represented a midpoint in the predicted craft work hours of Tables B-1 and B-3.
Culmulative Percent of Craft Work Hours for Projects Less Than $5M
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
15000- 20000 35000- 40000 55000- 60000 75000- 80000 <=0 95000- 100000 115000- 120000 135000- 140000 155000- 160000 175000- 180000 195000- 200000 215000- 220000 235000- 240000 255000- 260000 275000- 280000 295000- 300000 315000- 320000 335000- 340000 355000- 360000 375000- 380000 395000- 400000 415000- 420000 435000- 440000 455000- 460000 475000- 480000
Craft Work Hours
Figure B-5: Cumulative chart of craft work hours for projects less than $5M
223
Anchoring the Project Duration
Using a similar approach, Figure B-6 depicts the cumulative percent of project duration for projects less than $5M. The point where significant skew begins also is at the 85% level and corresponds to duration of approximately 14 months. This duration is reasonably consistent with Table B-2 and the export opinions from the Team. The midpoint of the range, 62.5 weeks, was converted to 14.2 month (1 month equals to 4.38 weeks).
Culmulative Percent of Project Duration (weeks) for Projects Less Than $5M
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
95- 100 <= 0 5- 10 15- 20 25- 30 35- 40 45- 50 55- 60 65- 70 75- 80 85- 90 105- 110 115- 120 125- 130 135- 140 145- 150 155- 160 165- 170 175- 180 185- 190 195- 200 205- 210 215- 220 225- 230 235- 240
Project Duration (weeks)
Figure B-6: Cumulative chart of Project Duration for projects less than $5M
224
Conclusion
In summary, the Team consensus was that the definition of a small project for this research effort is as follows:
A SMALL PROJECT IS DEFINED AS A PROJECT, WHICH HAS ONE OR MORE OF THE FOLLOWING ATTRIBUTES. 1. 2. 3. Total installed cost is between $100K and $5M Any duration of 14 months or less Any number of work-hours up to 100,000
In addition, the Team also agreed that: 4. Any level of complexity considering the following, design, construction, regulatory, owner coordination requirements or funding complexity. 5. The project does not require full time project management resources or does not require a significant percentage of the company's resources 6. Any nature of project (grass roots, modernization, addition or maintenance)
225
Appendix C
Company Bias Analysis
(MANOVA results of Owner Company)
Multivariate Analysis of Variance (MANOVA) is a common statistical technique to test the mean difference between groups for multiple dependent variables simultaneously. It is an extension of the ANOVA technique. The MANOVA technique is used here to test whether a particular company which has a large amount of projects submitted (OwnerCompany_A) introduced potential bias or not. Results for the difference in performance metrics are not statistically significant, indicating no mean difference in project performance. The
MANOVA test for practice use metrics is significant at alpha=0.05; However, further post-hoc tests reveals little difference. The test results reveal inconclusive evidence of a strong potential bias introduced by the company (OwnerCompany_A).
226
MANOVA Test for Differences in Performance Metrics
The SAS System
The GLM Procedure Class Level Information Class companycls Number of observations
NOTE:
Levels Values
OwnerCompany_A 2 Other 114 Observations with missing values will not be included in this analysis. Thus, only 89 observations can be used in this analysis.
The SAS System
The GLM Procedure Multivariate Analysis of Variance Characteristic Roots and Vectors of: E Inverse * H, where H = Type III SSCP Matrix for companycls E = Error SSCP Matrix Characteristic Vector V'EV=1 costgrow schdgrow costfact Characteristic Root Percent 100 0.39920686 1.03963207 0.0171042 -0.00643309 0 0.94932896 0 -0.10417894 -0.27130738 0 1.98066368 0 -0.31133387 -0.31451633 MANOVA Test Criteria and Exact F Statistics for the Hypothesis of No Overall companycls Effect H = Type III SSCP Matrix for companycls E = Error SSCP Matrix S=1 M=0.5 N=41.5 Value F Value
0.98318343 0.01681657 0.0171042 0.0171042 0.48 0.48 0.48 0.48
Statistic Wilks' Lambda Pillai's Trace Hotelling-Lawley Trace Roy's Greatest Root
Num DF
3 3 3 3
Den DF
85 85 85 85
Pr > F
0.6939 0.6939 0.6939 0.6939
227
Post-Hoc Tests
The SAS System
The GLM Procedure Tukey's Studentized Range (HSD) Test for costgrow
NOTE: This test controls the Type I experimentwise error rate, but it generally has a higher Type II error rate than REGWQ. 0.05 87 0.013671 2.8109 0.051 41.52809 Cell sizes are not equal.
Alpha Error Degrees of Freedom Error Mean Square Critical Value of Studentized Range Minimum Significant Difference Harmonic Mean of Cell Sizes
NOTE:
Tukey Grouping
A A A
Means with the same letter are not significantly different. Mean
-0.10104 -0.10894
N companycls
56 OwnerCompany_A 33 Other
The SAS System
The GLM Procedure Tukey's Studentized Range (HSD) Test for schdgrow
NOTE: This test controls the Type I experimentwise error rate, but it generally has a higher Type II error rate than REGWQ. 0.05 87 0.043979 2.8109 0.0915 41.52809 Cell sizes are not equal.
Alpha Error Degrees of Freedom Error Mean Square Critical Value of Studentized Range Minimum Significant Difference Harmonic Mean of Cell Sizes
NOTE:
Tukey Grouping
A A A
Means with the same letter are not significantly different. Mean
0.08188 0.03321
N companycls
56 OwnerCompany_A 33 Other
228
Post-Hoc Tests
The SAS System
The GLM Procedure Tukey's Studentized Range (HSD) Test for costfact
NOTE: This test controls the Type I experimentwise error rate, but it generally has a higher Type II error rate than REGWQ. 0.05 87 0.002752 2.8109 0.0229 41.52809 Cell sizes are not equal.
Alpha Error Degrees of Freedom Error Mean Square Critical Value of Studentized Range Minimum Significant Difference Harmonic Mean of Cell Sizes
NOTE:
Tukey Grouping
A A A
Means with the same letter are not significantly different. Mean
0.047 0.03803
N companycls
56 OwnerCompany_A 33 Other
229
MANOVA Test for Differences in Practice Use Metrics
The SAS System
The GLM Procedure Class Level Information Class companycls Number of observations
NOTE:
Levels Values
OwnerCompany_A 2 Other 114 Observations with missing values will not be included in this analysis. Thus, only 94 observations can be used in this analysis.
The SAS System
The GLM Procedure Multivariate Analysis of Variance Characteristic Roots and Vectors of: E Inverse * H, where H = Type III SSCP Matrix for companycls E = Error SSCP Matrix Characteristic Vector V'EV=1 smfep smdes smpro smcon smstu smorg Percent
100 0 0 0 0 0 0 0 0 0 -0.045 -0.052 0.003 -0.017 -0.009 -0.006 -0.011 0.055 -0.015 -0.010 -0.002 0.118 -0.020 0.028 -0.063 -0.019 -0.050 -0.009 -0.023 -0.022 -0.017 0.000 -0.008 0.006 0.075 0.001 -0.027 -0.005 -0.016 0.004 0.034 0.000 -0.020 0.013 0.004 -0.010 0.001 0.013 0.103 -0.032 -0.033 0.000 -0.010 -0.011 -0.004 0.007 0.068 -0.003 0.003 -0.008 -0.008 0.000 0.066 0.012 -0.002 0.000 -0.004 -0.005...