{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

Back_to_Basics_2 - A 1le I on I ll.1" Software...

Info icon This preview shows pages 1–5. Sign up to view the full content.

View Full Document Right Arrow Icon
Image of page 1

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 2
Image of page 3

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 4
Image of page 5
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: A 1le I on - I ll. ' - .1" Software Technology Support Center W \ I .. Home > CrossTalk Dec 2003 > Article CROSSTALKO The journal of Defense Software Engineering Dec 2003 Issue Back to the Basics: Measurement and Metrics Tim Perkins, Software Technology Support CenterfSAIl: Roald Peterson, SAII: Larry Smith, Software Technology Support Center Measuremenis and meincs are key tonis to understanding the behaviors, successes, and faiiures of our programs and projecis. This article highiighis the basic principies of measures and medics and encourages die reader to improve his or her use of these toais. The articfe is adapted fi'om {1]. According to Tom DeMarco, "You cannot control what you cannot measure" [2]. Imagine going on a road trip of over a thousand miles. This is easy because most of us really have done this several times. Now imagine that your car has no speedometer, no odometer, no fuel gauge, and no temperature indlcator. Imagine also that someone has removed the mile markers and road signs from all the roads between you and your destination. Just to complete the experiment, remove your watch. What was once a simple journey becomes an endless series of guesses, fraught with risks. How do you know where you are, how far you have gone, or how far you have to go? When do you gas the car? Should you stop here or try to make the next town before nightfall? You could break down, run out of gas, be stranded, take the wrong road, bypass your destination, or waste time trying to find your location and how to reach your destination. Clearly, some method of measuring certain indicators of progress is essential for achieving a goal. Imagine again going on a road trip. This time the cockplt of the car is fllled with instruments. In addition to what you have been accustomed to in the past, there are now the followlng gauges: Speed in feet and yards per second, and as a percentage of c (light speed). Oil pressure in millibars. Estimated time to deplete or recharge the battery. Fuel burn rate and fuel welght. Oil viscosity and transparency indicators. Antifreeze temperature and pressure. Engine efficiency. Alr conditioning system parameters (pressures, temperatures, efficienw). Elevatlon, rate of climb, heading, accelerometers for all directions. Indicators for distance and time to destination and from origln. Inside air temperatures for eight different locations in the car. Also, there are lnstruments to count how many cars pass, vibration levels, and sound pressure levels within and outside the car. There are weather indicators for outslde temperature, humidity, Vlsibility, cloud celling, ambient light level, true and relatlve wind speeds and directions, warning indicators for approaching storms and seismic activity, etc. Along the roads will be markers for every hundredth mile and signs announclng exits every quarter mile for five miles before an exit ls reached. Signs in five-mile-per-hour increments will announce speed changes. To some, thls may seem like a dream come true, at least the cockpit part. However, careful consideration Wlll soon reveal that the driver will be inundated and quickly overwhelmed with unnecessary, confusing data. Measurement, in itself, is no prescnption for achieving a goal. It can even make the goal unattalnable. Introduction Metrics are measurements of different aspects of an endeavor that help us determlne whether we are progresslng toward the goal of that endeavor. They are used extensively as management tools to provide some calculated, observable basis for making decisions. Some common metrics for projects include schedule deviation, remaining budget and expendlture rate, presence or absence of specific types of problems, and milestones achieved. Without some way to accurately track budget, time, and work progress, a project manager can only make declsions in the dark. Without a way to track errors and development progress, software development managers cannot make meaningful improvements in their processes. The more inadequate our metrics program, the closer we are to herding black cats in a dark room. The right metrics, used in the right way, are absolutely essential for project success. Too many metrics are used simply because they have been used for years, and people believe they might be useful [3]. Each metric should have a purpose, provlding support to a specific decislonmaking process. Leadership too often dlctates metrics. A team under the direction of leadership should develop them. Metrics should be used not only by leadership but also by all the various parts of an organizatlon or development team. Obvlously, not all metrics that are useful to managers are useful to the accountlng people or to developers. Metrics must be tailored to the users. The use of metrics should be defined by a program describing what metrics are needed, by whom, and how they are to be measured and calculated. The level of success or failure of your project Wlll depend in large part on your use or misuse of metrics - on how you plan, implement, and evaluate an overall metrics program. While this artlcle introduces and descrlbes key metrics ideas and processes and can point you In the right direction, it is recommended that you gain more insight, depth, and specific examples by downloading and readlng the material listed l|"l "Addltional Readlng" in the online version of this article at www.stsc.hill.af.milicrosstalk. Of particular value to Department of Defense users is [4] better known as the "Practical Software and System Measurement Guidebook," where a more specific and detailed terminology and methodology can be found. Process Description Metrics are not defined and used solely, but are part of an overall metrics program. This program should be based on the organlzation's goals and should be carefully planned, implemented, and regularly evaluated for effectiveness. The metrics program is used as a decision support tool. In relatlon to project management metrics, if the lnformation provided through a particular metric is not needed for determining status or direction of the project, lt is probably not needed at all. Process-related metrics, however, should not necessarily be dismissed so harshly since they indicate data useful l|"l lmprovmg performance across repeated applications. The role of the metrlcs program in the organlzatlon and its three major activities are shown in Figure l. Fioiect Management Urge nizslion Goals Men-ice Progra'n Figure 1: Metric; Program Cyde (Click on image above to shoW full-size version in pop-up window.) Developing a Metrics Program Plan Thefirst activity in developing a metrics program is planning. Metrics planning is usually based on the goaquuestionrmetric {GQM) paradigm developed by Victor Basili (see Figure 2). The GQM paradigm is based on thefollowing key concepts [3]: 1. Processes, including software development, program management, etc., have associated goals. 2. Each goal leads to one or more questions regarding the accomplishment of the goal. 3. Each question leads to one or more metrics needed to answer the question. 4. Each metric requires two or more measurements to produce the metric {e.g., miles per hour, budget spent vs. budget planned, temperature vs. operating limits, actual vs. predicted execution time, etc.). 5. Measurements are selected to provide data that will accurately produce the metric. GQM Define Goals Derive Questions Develop Metrics Figure 2: Basifi's Goal, Question, Metric Paradigm (Click on image above to show full-size version in pop-up window.) The planning process is comprised of the three sub-actIVities implementing the GQM paradigm and one that defines the data collection process. Each of these is discussed in the following sections. Table 1 shows two examples of goals and their related questions and metrics. Note that there could be one or more metrics associated with each question. As the initial list of questions and metrics is written and discussed, the goal is usually refined, which then causes a further refinement in the accompanying questions and metrics. Simona. III Producllm-h — m mmwmmm- Marin-W min-hum MMWM’MWW amylyflm? -susamiamuurmiaawmwmuomwmu -Agn mmlmlpnllamnlmm. pmdxlx. pmdl-tlrelnrnnll, .1 mung-mum summon: mmflhhpml- dammlmf - Tm» simulate 1M. mam-i leaner): m, vuamca mum to many. “ammonium tie. nwwewrofllvwfimm. mum, m. mlmwmmmwm wulmmmammpnnnmmm Io Ila? uh" ‘ll'lll'!’ “madman-y Inn -Cieinne|auwym concatenated. sic. MJaMmewrw-m anmmmmm mlfl-BWNIW alumnus-mm? - "mm-www.maimimm. mum mmhmnm.pockolmouac1flemy. an. Table 1: Goals and Their Refated Questions and Metrics (Click on image above to show fullisize version in poprup window.) Define Goals Planning begins With well defined, validated goals. Goals should be chosen and worded in such a way that they are verifiable; that is, their accomplishment can be measured or observed in some way. Goals such as meeting a specific delivery schedule are easily observable. Requirements stating "software shall be of high quality" are highly subjective and need further definition before they can be used as valid goals. You may have to refine or even derive your own goals from loosely written project objectives. The selection or acceptance of project goals Will determine how you manage your project, and where you put your emphasis. Goals should meet the folloWing criteria: 3 They should support the successful accomplishment of the project's overall or system-level goals. 0 They should be verifiable, or measurable in some way. I They should be defined in enough detail to be unambiguous. Derive Questions Each goal should evoke questions about how its accomplishment can be measured. For example, completing a project within a certain budget may evoke questions such as these: What is my total budget? HoW much of my budget is left?I What is my current spending rate? Am 1 within the limits of my spending plan? Goals related to software time, size, quality, or reliability constraints would evoke different questions. It should be remembered that different levels and groups within the organization might require different information to measure the progress in which they are interested. Questions should be carefully selected and refined to support the previously defined project goals. Questions should exhibit the following traits: 0 Questions only elicit information that indicates progress toward or completion of a specific goal. 0 Questions can be answered by providing specific information. {They are unambiguous.) 0 Questions ask all the information needed to determine progress or completion of the goal. Dnce questions have been derived that elicit only the complete set of information needed to determine progress, metrics must be developed that will provide that information. Develop Hetrirs Metrics are the information needed to answer the derived questions. Each question can be answered by one or more metrics. These metrics are defined and assoCiated with their appropriate questions and goals. Each metric requires two or more measurements. Measurements are those data that must be collected and analyzed to produce the metric. Question QI-Iestlon Question Figure 3: Goal, Question, Meb’fcs Examples (Click on image above to show full-size version in pop-up window.) Measurements are selected that Will provide the necessary information with the least impact to the project work‘flow. Figure 3 summarizes the process of turning measurements into final s‘ffih IR. Choosing measures is a critical and nontriVIal step. Measurements that require too much effort or time can be counterproductive and should be avoided. Remember, just because something can be measured does not mean it should be. An in-depth introduction to measurements, "Goal-Driven Software Measurement - A Guidebook," has been published by the Software Engineering Institute and is available as a free download [5]. In addition to choosing what type of data to collect or measure, the methods of processing or analysis must also be defined in this step. How do you turn the measurements into a meaningful metric? How does the metric then answer the question? The analysis method should be carefully documented. Do not assume that it is obvious. This activity is complete when you know exactly what type of data you are going to collect (what you are going to measure and in what units), how you are going to turn that data into metrics (analysis methods), and in what form (units, charts, colors, etc.) the metrics will be delivered. Define the Collection Process The final step of the metrics planning process is to determine how the metrics will be collected. At a minimum, this part of the plan should include the following: What data is to be collected? What Will be the source of the data? How is it to be measured? who will perform the measurement? How frequently should the data be collected? Who will the derived metrics be delivered to, and in what format? Implementing a Metrics Program A good rule of thumb to follow when starting a measurement program is to keep the number of measurements between five and 10. If the metrics program is well planned, implementing the program should be reduced to simply following the plan. There are four activities in the metrics implementation cycle, shown in Figure 4 [6]. "—Fl Collect Data. H Validate Dala H Denve Metrics I—)| Make Denisions In. HERE? ‘3‘. 5PM? Figure 4: Mab‘r'cs Implementation Cyde (Click on image above to show full-size version in pop-up window.) Data is collected at specific intervals according to the plan. Data is then validated by examining it to ensure it is the result of accurate measurements, and that the data collection is consistent among members of the group if more than one individual is collecting it. In other words, is it being measured in the same way, at the same time, etc.1I Once the data is determined to be valid, the metrics are derived by analyZing the data as documented in the metrics program plan. Metrics are then delivered to appropriate individuals and groups for evaluation and decision-making activities. This process is repeated until the project is complete. Evaluating a Metrics Program It is likely that a metrics program will not be perfect in its first iteration. Soon after its initial implementation and at regular intervals after that, the metrics program should be evaluated to determine if it is meeting the needs of the metrics users, and if its implementation is flowmg smoothly. If metrics prove to be insufficient or superfluous, the program plan should be modified to provide the necessary information and remove any unneeded activity. The objective of a metrics program is to provide sufficient information to support project success while keeping the metrics program as simple and unobtrusive as possible. The following are areas that should be considered when reviewing a metrics program: Adequacy of current metrics. Super‘fluity of any metrics or measures. Interference of measurements With project work. Accuracy of analysis results. Data collection intervals. Simplification of the metrics program. Changes in project or organization goals. Metrics Repository A final consideration is establishing a metrics repository where metrics history is kept for future projects. The availability of past metrics data can be a gold mine of information for calibration, planning estimates, benchmarking, process improvement, calculating return on investment, etc. At a minimum, the repository should store the following: 0 Description of projects and their objectives. 0 Metrics used. 0 Reasons for using the various metrics. 0 Actual metrics collected over the life of each project. 0 Data indicating the effectiveness of the metrics used. Measurement and Metrics Checklist This checklist is provided to assist you in developing a metrics program, and in defining and using metrics. If you cannot answer a question affirmatively, you should carefully examine the situation and take appropriate action. The checklist items are divided into three areas: developing, implementing, and reVIewmg a metrics program. Developing a Metrics Program Is your use of metrics based on a documented metrics program plan? Are you using the GQM paradigm in developing your metrics? Are your metrics based on measurable or verifiable project goals? Do your goals support the overall system- level goals? Are your goals well defined and unambiguous? Does each question elicit only information that indicates progress toward or completion a specific goal? Can questions be answered by proViding specific information? {Is it unambiguous?) Do the questions ask for all the information needed to determine progress or completion of the goal? I Is each metric required for specific decision-making activities? l Is each metric derived from two or more measurements (e.g., remaining budget vs. schedule)? Have you documented the analysis methods used to calculate the metrics? Have you defined those measures needed to provide the metrics? Have you defined the collection process (i.e., what, how, who, when, how often, etc.)? _..,_ _g_g_g_g_g etrics Program Implementation Does your implementation follow the metrics program plan? Is data collected the same way each time it is collected? Are documented analysis methods followed when calculating metrics? Are metrics delivered in a timely manner to those who need them? Are metrics being used in the decision- making process? Metrics Program Evaluation Are the metrics sufficient? Are all metrics or measures required, that is, non-superfluous? Are measurements allowing project work to continue without interference? Does the analysis produce accurate results? Is the data collection interval appropriate? as simple as it can be while r _ i____ __u.£._4 I. _4__ ._.._ i'ig adequ t ? ___4_I._ g. _ _ _._._. our nos me memes program been niuuineu Lu auequateiy clL'L'UHIHIiJiJalLe any prujeot or organizational goal changes?I References 1. Software Technology Support Center. Condensed Version {4.0) of Guidelines for Successful Acquisition and Management of Software-Intensive Systems. Hill Air Force Base, Utah: Software Technology Support Center, Feb. 2003. DeMarco, Tom. Controlling Software Projects. New York: Yourden Press, 1932. Perkins, Timothy K. "The NinenStep Metrics Program." CrossTalk, Feb. 200]. www.stsc.hill.af.milr'crosstalkf2001f02r'perkins.html. 4. Bailey, Elizabeth, et al. Practical Software Measurement: A Foundation for Objective Project Management Ver. 4.0b1. Severna Park, MD: Practical Software and Systems Measurement, Mar. 2003 www. psmsc.comifPSMGuide.asp under "Products." 5. Park, Robert E., et al. Goal-Driven Software Measurement - A Guidebook. Pittsburgh, PA: Software Engineering Institute, Aug. 1996 www.sei.cmu.edufgublications 1document_s196.rej:_iort_sfi96.hb.002.html. 6. Augustine, Thomas, et al. "An Effective Metrics Process Model." CrossTalk June 1999 www.stsc.hi||.af.mi|,r_'crossta||-_<£1999fi06,r_'augustine.asp. Additional Reading 1. Budlong, Faye C., and Judi A. Peterson. "Software Metrics Capability Evaluation Methodology and Implementation." CrossTalk Jan. 1996 www.stsc.hill.af.milficrosstalk. 2. Carlton, Anita D., et al. "Software Measurement for DoD Systems: Recommendations for Initial Core Measures." Pittsburgh, PA: Software Engineering Institute, Sept. 1992 www.sei.cmu.edufpublicationsldocumentleZJeports 192.tr.019.htm|. 3. Fleming, Quentin W., and Joel M. Koppelman. "Earned Value Project Management: A Powerful Tool for Software Projects." CrossTaIk July 1993 www.stsc.hi||.af.mi| crosstalk. 4. Florac, William A. "Software Quality Measurement: A Framework for Counting Problems and Defects." Pittsburgh, PA: Software Engineering Institute, Sept. 1992 www.sei.cmu.edufpublications/documents/92.repor‘tsu’92.tr.022.html. 5. Florac, William...
View Full Document

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern