COMS309Measurement

COMS309Measurement - Measurement COMS 309 David Weiss...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Measurement COMS 309 David Weiss Fall, 2010 weiss@cs.iastate.edu Problems in soBware development •  Knowing what the customers want; knowing the requirements •  PredicJng Jme to develop •  PredicJng resources needed to develop •  Managing change •  Knowing when you’re done •  Designing to enable distribuJon of work •  CoordinaJng work •  Knowing what the value is (how much will I make) •  Tracking Jme, resources, quality, producJvity, effecJveness, etc. COMS 309 Weiss Fall 2010 Measurement 2 Solving Problems and Knowing It Requires Measurement •  What should we measure? •  How should we develop the measures? •  How do we get the data for the measurement? •  How do we analyze the data? COMS 309 Weiss Fall 2010 Measurement 3 Goal, QuesJon, Metric: A Measurement Paradigm (1) •  Establish the goals of the project –  Examples: •  Keep customers happy •  Generate members of a product line three Jmes as fast as previously •  Learn methods for, advantages of, and difficulJes of working in a team •  Define quesJons that, when answered, establish progress towards the goal •  Define measures that, when computed, answer the quesJons and thereby establish progress towards the goal •  GQM COMS 309 Weiss Fall 2010 Measurement 4 Goal, QuesJon, Metric: A Measurement Paradigm (2) •  Define quesJons that, when answered, establish progress towards the goal –  Examples: •  Keep customers happy –  How many customer complaints did we receive last month? Six months ago? –  How many failures per installed product were there last month? •  Generate members of a product line three Jmes as fast as previously –  How long did it take to develop the last 5 products using the old methods? –  How long does it take to develop a product now? •  Learn methods for, advantages of, and difficulJes of working in a team –  What were the primary difficulJes each team had? –  Which team produced the best products? The worst? What were the differences in team dynamics between the two? COMS 309 Weiss Fall 2010 Measurement 5 Goal, QuesJon, Metric: A Measurement Paradigm (3) •  Define measures that, when computed, answer the quesJons and thereby establish progress towards the goal –  Examples: •  Keep customers happy –  How many customer complaints did we receive last month? Six months ago? »  Measure: Customer complaints per delivered system per month –  How many failures per installed product were there last month? »  Measure: Failures per installed product per month as a funcJon of Jme •  Generate members of a product line three Jmes as fast as previously –  How long did it take to develop the last 5 products using the old methods? »  Measure: Time to develop a product, as a funcJon of Jme –  How long does it take to develop a product now? »  Measure: Time to develop a product, as a funcJon of Jme •  Learn methods for, advantages of, and difficulJes of working in a team –  What were the primary difficulJes each team had? »  Measure: difficulJes described by teams and observed by instructors, prioriJzed –  Which team produced the best products? The worst? What were the differences in team dynamics between the two? »  Measure: Usefulness of products, as judged by instructors and outside observers »  Measure: Difficulty of use of products, on a scale of 1 ­5, as evaluated by instructors »  … COMS 309 Weiss Fall 2010 Measurement 6 PredicJng SoBware Development (50 sampled projects) Project Es%mates At Gate 1 200% Project Es%mates at Gate 2 Median 150% Rela%ve Schedule Range 100% 75% 50% Plans and Requirements Feasibility Product Design Detailed Design Development and Test Launch COMS 309 Weiss Fall 2010 Measurement 7 How Does ProducJvity Change Over Time? (1) Developer Churn for a Sample Project COMS 309 Weiss Fall 2010 Measurement Project Moved Offshore To Inexperienced Group 8 How Does ProducJvity Change Over Time? (2) Produc%vity Trend for a Sample Project Core Group: Developers who made 80% of changes Project Moved Offshore To Inexperienced Group Interval Quality • Probability that a customer observes a failure within one, three, and six months aBer installaJon •  Drawback –  does not account for the proximity to launch •  Significant differences are marked with *, **, and *** •  PrioriJes changed from Jme ­ to ­market to quality Release COMS 309 Weiss Fall 2010 Measurement 10 Goals •  Learn methods for, advantages of and difficulJes of working in a team, similar to industry experience •  Understand why soBware development organizaJons establish and ajempt to follow well ­defined soBware lifecycle processes. •  Produce a working applicaJon •  Students will form their own teams; each team will have a set of defined roles, such as systems engineer, architect, developer, tester, and project manager. Students may play more than one role in a team. •  Each team will give a progress report midway through the semester, and each team will demonstrate its product at the end of the semester and will create a poster describing the product and the team’s learnings as a result of developing the product. •  Each team will hold a retrospecJve at the end of the semester and produce a retrospecJve report. •  Have some fun COMS 309 Weiss Fall 2010 Measurement 11 GQM: Data CollecJon •  Data collecJon should be –  Unobtrusive: don’t add overhead to exisJng jobs –  Valid: data can be used to answer the quesJon(s) of interest –  Verifiable: accuracy of the data can be determined and esJmated (may need to sample) –  Repeatable: different people collect the same data and get the same results –  QuanJtaJve: objecJve, numerical data –  QualitaJve: opinion surveys, interviews •  Types of data collecJon •  It’s never that simple COMS 309 Weiss Fall 2010 Measurement 12 Data collecJon: an example of a qualitaJve approach •  Goal: When new developers join a project, train them effecJvely and efficiently •  QuesJons: –  –  –  –  What does a new developer need to learn when joining a project? What help was available to new developers? What resources did new developers have to reference? How did new developers resolve problems? •  Approach: –  Design interview quesJons, focus on developers’ involvement: –  Sample people for interviews: 3 developers per project, focus on those who made most changes to shared files –  Explain our purpose before the interview, and promise privacy of their idenJty –  Conduct the interviews through telephone conferences, one project each Jme with two colleagues helping f2f COMS 309 Weiss Fall 2010 Measurement 13 Interview QuesJonnaire, Part 1: Involvement in organizaJon and project a. When did you graduate from the university? b. What was your former job before you joined the company? c. Did you take any training courses when you joined the project? How long did they last? What were the contents of the training course(s)? d. What does your project do? Which part do you work on, such as developing a part of a system, fixing bugs (current engineering), and so on? e. How do you receive your assignment (supervisor, self)? Why were you assigned to work on this (technical background, your interest, availability)? f. What did you have to learn for your assignment, e.g., language, architecture, how to use version control/ problem tracking, how to build the project specific development environment, how to write a test, how to understand the exisJng code? What is the most difficult part? g. What techniques did you use to understand the exisJng code? What factors influenced your understanding, such as code style, code comments, nobody to ask, etc? h. Does your project require code style/code comments? i. What help was available, such as reading documentaJon, asking a specified mentor, asking peers, when you came into the project? How has that changed? j. What kinds of support can you provide (quesJons, help with code changes, documentaJon, build, test, code understanding)? l. Please give some examples of parJcularly useful help you got from (or provided to) others. m. What’s the biggest barrier when you get/provide help? j. What kind of documents do you have, such as development document, requirements document, design document, test document? What are the most useful documents for you? k. What did you have to learn by yourself, not included in the documents, and difficult to get from any others? Please give an example. COMS 309 Weiss Fall 2010 Measurement 14 Interview QuesJonnaire, Part 2: Factors that affect the outcome of succession a. How many people are in your project? Who is your mentor? b. Who are the people who are the people whom you are most likely to ask quesJons? How many people ask you quesJons or need your support, how many people do you ask quesJons or need support from? c. How many people from your project are on your site and how many are on another site? d. What’s the most important thing you learned from the people with whom you worked? (coding style, how to build/test, interface, ..) e. How did you learn that, such as from discussions, reading his/her code, observing development habits, etc.? f. What kinds of meeJng do you have (size: project, subgroup, one ­on ­one, type: progress, coordinaJon, resolve a specific issue)? g. How is the meeJng conducted (Is there any agenda before the meeJng? Who hosts the meeJng? What do you discuss and decide in the meeJng?) h. How oBen do you have meeJngs? i. What helps most to be more producJve in the project? (code comments, documentaJon, training, f2f meeJngs...) j. What could be improved to make you more producJve? (bejer training, knowing who can help you, bejer code, ...) COMS 309 Weiss Fall 2010 Measurement 15 Interview QuesJonnaire, Part 3: AddiJonal InformaJon a. What’s most difficult for you? b. What would make you happier? c. Is there anything else we should have asked but did not? COMS 309 Weiss Fall 2010 Measurement 16 Data collecJon: an example of a quanJtaJve approach •  Goal: Enhance developers’ learning when joining projects •  QuesJons: –  What does learning bejer mean? •  Higher producJvity –  What is the developer’s producJvity when joining the project –  What is the developer’s producJvity as a funcJon of Jme in the project? –  How do developers learn? •  Through pracJce and over Jme –  What is the developer’s producJvity as a funcJon of Jme in the project? COMS 309 Weiss Fall 2010 Measurement 17 Measure producJvity as changes •  SoBware is created by making changes to it –  A delta is a single checkin (ci/commit/edput) represenJng an atomic modificaJon of a single file with the following ajributes •  File, Date, Developer, Comment –  Other ajributes that oBen can be derived: •  Size (number of lines added,deleted) •  Lead Jme (interval from start to compleJon) •  Purpose (Fix/New Feature) •  Approach –  Use project’s repositories of change data to model (explain and predict) phenomena in soBware projects and to create tools that improve soBware producJvity/quality/lead Jmes COMS 309 Weiss Fall 2010 Measurement 18 Change Hierarchy COMS 309 Weiss Fall 2010 Measurement 19 Measure learning as proficiency at making changes •  Developer producJvity – number of changes per staff ­month •  Learning experience •  tenure, i.e., the months from hiring day •  pracJce, i.e., the changes the developer has made up to that month COMS 309 Weiss Fall 2010 Measurement 20 Developers learn through pracJce and over Jme •  X: months since joining the company •  Y: changes per developer that month (average over developers who made changes) A B C COMS 309 Weiss Fall 2010 Measurement 21 An example of GQM in soBware engineering •  AT&T used GQM to help assess their inspecJon process (1994) •  Goal: Make inspecJons efficient •  QuesJons and metrics: –  How much does the inspecJon process cost? •  Average effort per KLOC •  Percentage of re ­inspecJons –  How much calendar Jme does the inspecJon process take? •  Average effort per KLOC •  Total KLOC inspected COMS 309 Weiss Fall 2010 Measurement 22 Effort and producJvity in TrusJe •  Goal: understand how project is going •  QuesJons and metrics: – How much effort is spent per month/week? •  developers/month or week – What is the output? •  ProducJvity (changes/Jme unit) COMS 309 Weiss Fall 2010 Measurement 23 Measuring Change •  SoBware is created by changes •  Changes are tracked COMS 309 Weiss Fall 2010 Measurement 24 Effort and ProducJvity COMS 309 Weiss Fall 2010 Measurement 25 Defect predicJon •  Goal: Predict bug fixes, e.g, in order to plan effort •  QuesJons: What are factors associated with bug fixes? •  Metric: Use new feature mrs to predict bug fixes, both number and trend AssumpJon: –  A change may introduce defect(s) with some delay –  The new ­feature changes are dictated by the business environment and are assumed as given –  New feature mrs in the past releases –  New feature mrs in the current release COMS 309 Weiss Fall 2010 Measurement 26 Measuring The Effect of PLE In Legacy Systems That Have Been Reengineered •  Environment: Large legacy systems with many domains •  Goal: Measure effort and Jme per change before and aBer a domain is reengineered •  Data Needed –  Change history •  Type, Time, Effort, Developers •  Tags in the code that indicate reengineering •  Lucent results –  Effort (or Jme) improvement of about a factor of 3 COMS 309 Weiss Fall 2010 Measurement 27 Interval ReducJon Percent 100 80 60 40 20 0 A B C D Previous Interval Reduced Interval COMS 309 Weiss Fall 2010 Measurement 28 Obstacles to measurement •  Lack of focus on soBware measurement –  Low priority except in emergencies –  Need for immediate results (short Jme horizon) –  Lack of resources for measurement/improvement –  Difficulty of obtaining, understanding, and validaJng data –  MulJple stakeholders (developer/support/product management) –  Difficulty of comparison among projects, even earlier versions within the same project •  Establish infrastructure for data collecJon and analysis? –  Dashboards –  Automated data collecJon and analysis COMS 309 Weiss Fall 2010 Measurement 29 "When you can measure what you are speaking about, and express it in numbers, you know something about it, but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsa%sfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science.” William Thomson, Lord Kelvin COMS 309 Weiss Fall 2010 Measurement 30 Summary •  Need to measure to understand progress towards goals •  GQM is a useful measurement paradigm •  QualitaJve and quanJtaJve measures •  Unobtrusive, automated data collecJon •  Meaningful measurement is difficult •  Comparison among projects is difficult COMS 309 Weiss Fall 2010 Measurement 31 Terminology •  Goal, QuesJon, Metric (GQM) COMS 309 Weiss Fall 2010 Measurement 32 Exercise 3: Measuring Product Lines •  IdenJfy two goals of soBware product line engineering •  For each goal that you idenJfied, propose three quesJons that help gauge progress towards the goal •  For each quesJon that you proposed, define one measure that helps answer the quesJon •  For each measure, sketch or tabulate results that you might expect to get, i.e., the expected outcomes COMS 309 Weiss Fall 2010 Measurement 33 COMS 309 Weiss Fall 2010 Measurement 34 Expected Outcomes COMS 309 Weiss Fall 2010 Measurement 35 ...
View Full Document

Ask a homework question - tutors are online