JTC1/SC7 Untitled
ISO/IEC N1430 1995-08-08 TITLE: CD 14598-1.2 Software Product Evaluation Part 1: General Overview WG6 07.13.02.01 2nd CD SC7 N1137, N1164 and Resolutions 397 and 408 Ballots and comments must be received at the SC7 Secretariat no later than 1995-11-22. E-Mail responses would be appreciated.
SOURCE: WORK ITEM: STATUS: REFERENCE: ACTION:
Attached is a compressed (ZIP) archive of Poscript (.ps) and MS Word for Windows 2.0 (.doc) files. Size: ziped 81 033 bytes 7n1430_l.ps 7n1430_a.ps 7n1430_c.doc 136 941 bytes (Letter format) 137 034 bytes (A4 format) 122 368 bytes (cover page + ballot)
decompressed:
Page 1
ISO/IEC JTC1/SC7/WG6 N
Evaluation and Metrics
TITLE: ISO/IEC draft CD 14598-1 Information Technology - Software Product Evaluation Part 1: General Overview 20 July 1995 JTC1/SC7/WG6 Project 7.13.3 Version 6.4 - Incorporating comments following Brisbane meeting
DATE: SOURCE: WORK ITEM: STATUS: DOCUMENT TYPE: EDITOR:
2nd CD Prof. Motoei AZUMA Department of Industrial Eng. and Management Waseda University 3-4-1, Okubo, Shinjuku-ku, Tokyo 169, Japan FAX: [phone] [e-mail] Nigel BEVAN Div. of Information Technology and Computing National Physical Laboratory Teddington, Middx. TW11 0LW, United Kingdom FAX: [phone] [e-mail] Andrew CHRUSCICKI Rome Laboratory Software Engineering Branch RL/C3CB 525 Brooks Rd. Griffis AFB, NY 13441-4505 USA FAX: [phone] [e-mail]
CO-EDITOR:
CO-EDITOR:
-1-
ISO/IEC CD 14598-1
v6.4 20 July 1995
Contents
0. FOREWORD 1. SCOPE 2. NORMATIVE REFERENCES 3. CONFORMANCE 4. DEFINITIONS 5. OVERVIEW OF SOFTWARE PRODUCT EVALUATION STANDARDS
5.1 Structure of standards 5.2 Evaluation process 5.2.1 Process for developers 5.2.2 Process for acquirers 5.2.3 Process for evaluators 5.3 Support for evaluation 5.3.1 Planning and Management 5.3.2 Evaluation modules 5.4 Software Quality Characteristics and Metrics 5.4.1 Quality Characteristics and Subcharacteristics 5.4.2 External metrics 5.4.3 Internal metrics
4 6 6 7 7 10
10 10 10 10 10 10 11 11 11 12 12 12
6. EVALUATION PROCESS 7. ESTABLISH EVALUATION REQUIREMENTS
7.1 Establish the purpose of evaluation 7.1.1 General 7.1.2 Acquisition 7.1.3 Development 7.1.4 Supply 7.1.5 Operation 7.1.6 Maintenance 7.2 Identify types of product(s) to be evaluated 7.3 Specify quality model
12 13
13 13 14 14 14 15 15 15 17
8. SPECIFICATION OF THE EVALUATION
8.1 Select metrics 8.1.1 Types of measurements 8.1.2 Requirements for measurements
18
18 19 19
-2-
v6.4 20 July 1995 8.2 Establish rating levels for metrics 8.3 Establish criteria for assessment
ISO/IEC CD 14598-1 19 20
9. DESIGN OF THE EVALUATION
9.1 Produce evaluation plan
20
20
10. EXECUTION OF THE EVALUATION
10.1 Take measures 10.2 Compare with criteria 10.3 Assess results
20
20 21 21
11. SUPPORTING PROCESSES
21
ANNEX A (INFORMATIVE) INFORMATIVE REFERENCES AND BIBLIOGRAPHY 22 0.
-3-
ISO/IEC CD 14598-1
v6.4 20 July 1995
Foreword
ISO (the International Organisation for Standardisation) and IEC (the International Electrotechnical Commission) form the specialised system for worldwide standardisation. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organisation to deal with particular fields of mutual interest. Other international organisations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information technology, ISO and IEC have established a joint technical committee ISO/IEC JTC 1. Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as an International Standard requires approval by at least 75% of the national bodies casting a vote. International Standard ISO/IEC 14598-1 was prepared by Joint Technical Committee ISO/IEC JTC1 Information Technology. ISO/IEC 14598 consists of the following parts under the general title Information Technology Software product evaluation Part 1: General overview Part 2: Planning and management Part 3: Process for Developers Part 4: Process for Acquirers Part 5: Process for Evaluators Part 6: Evaluation modules
-4-
v6.4 20 July 1995
ISO/IEC CD 14598-1
Introduction
This part of ISO/IEC 14598 provides general requirements and a guide to the ISO/IEC 14598 series of standards and technical reports which deal with software product quality. ISO/IEC 14598 can be used by acquirers, users, producers and independent professional evaluators who wish to evaluate the quality of software products. As the use of information technology grows, the number of critical computer systems also grows. Such systems include for example, security critical, life critical, economically critical and safety critical systems. The quality of software in these systems is particularly important because software faults may lead to serious consequences. Throughout the history of software engineering, software quality improvement has been a most important goal. The evaluation of software product quality is vital to both the acquisition and production of software which meets quality requirements. The relative importance of the various characteristics of software quality depends on the mission or objectives of the system of which it is a part; software products need to be evaluated to decide whether relevant quality characteristics meet the requirements of the system. The essential parts of software quality evaluation are a quality model, the method of evaluation, software measurement, and supporting tools. To develop good software, quality requirements should be specified, the software quality assurance process should be planned, implemented and controlled, and both intermediate products and end products should be evaluated. To achieve objective software quality evaluations, the quality attributes of the software should be measured using validated metrics. The term "metric" has been used in many senses in software engineering publications. In this standard it is defined as a quantitative scale and method which can be used for measurement. The word "measure" is used to refer to the result of a measurement. The ISO/IEC 14598 series of standards give methods for measurement, assessment and evaluation of software product quality. They describe neither methods for evaluating software production processes nor methods for cost prediction (software product quality measurements may, of course, be used for both these purposes).
-5-
COMMITTEE DRAFT
ISO/IEC 14598-1
Information Technology - Software Product Evaluation Part 1: General Overview
1. Scope
This part of ISO/IEC 14598 introduces the other parts. It contains general requirements for specification and evaluation of software quality, defines the technical terms used in the other parts and clarifies the general concepts. Additionally, it provides a framework for evaluating the quality of all types of software product and states the requirements for methods of software product measurement and evaluation. This standard is intended for use by developers, including managers, requirement analysts, software designers, implementors and quality assurance staff; by users, including managers, engineering staff and end users; by third party software evaluators and by those responsible for evaluation technology development and standardisation.
NOTE - Much of the guidance in this series of standards is not specific for software, but is also applicable to other complex products.
2. Normative References
The following standards contain provisions which, through reference in this text, constitute provisions of this part of ISO/IEC 14598. At the time of publication, the editions indicated were valid. All standards are subject to revision, and parties to agreements based on this part of ISO/IEC 14598 are encouraged to investigate the possibility of applying the most recent editions of the standards indicated below. Members of IEC and ISO maintain registers of currently valid International Standards. ISO/IEC 2382-1:1993 Data processing - Vocabulary - Part 1: Fundamental terms ISO/IEC 2382-20:1990 Information Processing Vocabulary - Part 20: System development ISO 8402:(new) Quality Vocabulary. ISO/IEC 9126:(new) Information Technology - Software quality characteristics and metrics - Part 1: Quality characteristics ISO/IEC 9126:(new) Information Technology - Software quality characteristics and metrics - Part 2: External metrics ISO/IEC 9126:(new) Information Technology - Software quality characteristics and metrics - Part 3: Internal metrics ISO/IEC 12207 (1995) Information Technology - Software life-cycle processes ISO/IEC 14598-2:(new) Information Technology - Software product evaluation - Part 2: Planning and management ISO/IEC 14598-3:(new) Information Technology - Software product evaluation - Part 3: Process for development
-6-
v6.4 20 July 1995
ISO/IEC CD 14598-1
ISO/IEC 14598-4:(new) Information Technology - Software product evaluation - Part 4: Process for acquisition ISO/IEC 14598-5:(new) Information Technology - Software product evaluation - Part 5: Process for evaluation ISO/IEC 14598-6:(new) Information Technology - Software product evaluation - Part 6: Evaluation modules [Editor's Note - any of the above which are not normative references in the final text will be moved to Annex A or deleted.]
3. Conformance
Specification and evaluation of software conforms to this part of ISO 14598 if it meets the relevant requirements of clauses 8.1.2 (requirements for measurements), and 8.3 (criteria for assessment).
4. Definitions
For the purposes of all parts of ISO/IEC 14598, the following definitions apply. 4. 1 quality: The totality of characteristics of an entity that bear on its ability to satisfy stated and implied needs. [New ISO 8402]
NOTE - In a contractual environment, or in a regulated environment, such as the nuclear safety field, needs are specified, whereas in other environments, implied needs should be identified and defined (ISO 8402 : 1994, note 1).
4. 2 implied needs: Needs that may not have been stated but are actual needs when the entity is used in particular conditions.
NOTE - Implied needs are real needs which may not have been documented.
4. 3 software: All or part of the programs, procedures, rules, and associated documentation of an information processing system. ( ISO/IEC 2382-1 : 1993)
NOTE - Software is an intellectual creation that is independent of the medium on which it is recorded.
4. 4 software product: The set of computer programs, procedures, and possibly associated documentation and data designated for delivery to a user. [ISO/IEC 12207-1]
NOTE - products include intermediate products, and products intended for users such as developers and maintainers.
4. 5 user: An individual that uses the software product to perform a specific function.
NOTE - Users may include operators, recipients of the results of the software, or developers or maintainers of software.
4. 6 quality model: The set of characteristics and the relationships between them which provide the basis for specifying quality requirements and evaluating quality. 4. 7 external quality: The extent to which a product satisfies stated and implied needs when used under specified conditions.
-7-
ISO/IEC CD 14598-1
v6.4 20 July 1995
4. 8 internal quality: The totality of attributes of a product that determine its ability to satisfy stated and implied needs when used under specified conditions.
NOTE - The term internal quality, used in this standard to contrast with external quality, has essentially the same meaning as quality in ISO 8402. The term attribute is used as the term characteristic is used in a more specific sense in ISO 9126.
4. 9 quality in use: The extent to which a product can be used by specified users to meet their needs to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. 4. 10 attribute: A measurable physical or abstract property of an entity.
NOTE - Software quality characteristics are defined in ISO 9126-1 in terms of attributes of the software.
4. 11 measurement: The process of assigning a number or category to an entity to describe an attribute of that entity.
NOTE - "category" is used to denote qualitative measures of attributes. For example, some important attributes of software products, e.g. the language of a source program (ADA, C, COBOL, etc.) are qualitative.
4. 12 measure: The number or category assigned to an attribute of an entity by making a measurement. 4. 13 direct measure: A measure of an attribute that does not depend upon a measure of any other attribute 4. 14 indirect measure: A measure of an attribute that is derived from measures of one or more other attributes 4. 15 internal measure: A measure derived from the product itself, either direct or indirect; it is not derived from measures of the behaviour of the system of which it is a part.
NOTE - Lines of code, complexity, the number of faults found in a walk through and the Fog Index are all internal measures made on the product itself.
4. 16 external measure: An indirect measure of a product derived from measures of the behaviour of the system of which it is a part.
NOTES 1. For example, measures of a computer system executing a software product can be used as indirect measures of the software product. 2. The number of faults found during testing is an external measure of the number of faults in the program because the number of faults are counted during the operation of a computer system running the program to find the number of faults in the code. 3. External measures can be used to evaluate quality attributes closer to the ultimate objectives of the design.
-8-
v6.4 20 July 1995
ISO/IEC CD 14598-1
4. 17 measurement scale: A scale that constrains the type of data analysis that can be performed on it: nominal which corresponds to a set of categories; ordinal which corresponds to an ordered set of scale points; interval (which corresponds to an ordered scale with equidistant scale points; ratio which not only has equidistant scale point but also possess an absolute zero. 4. 18 rating: The action of mapping the measured value to the appropriate rating level. Used to determine the rating level associated with the software for a specific quality characteristic. 4. 19 rating level: A scale point on an ordinal scale which is anchored to a range of values on another ordinal, interval or ratio scale measure.
NOTES 1. The rating level enables software to be classified (rated) in accordance with the stated or implied needs. 2. Appropriate rating levels may be associated with the different views of quality i.e. Users', Managers' or Developers'. 3. These rating levels are different from the "grades" defined in ISO 8402.
4. 20 indicator: A measure of a characteristic that can be used to estimate or predict another measure of the same or a different characteristic.
NOTE - Indicators may be used both to estimate software quality attributes and to estimate attributes of the production process.
4. 21 metric: A quantitative scale and method which can be used for measurement. 4. 22 quality evaluation: Systematic examination of the extent to which an entity is capable of fulfilling specified requirements. [ISO 8402] 4. 23 verification: Confirmation by examination and provision of objective evidence that specified requirements have been fulfilled.
NOTES 1. In design and development, verification concerns the process of examining the result of a given activity to determine conformity with the stated requirement for that activity. 2. "Verified" is used to designate the corresponding status.
[ISO 8402: 1994, 2.17] 4. 24 validation: Confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use are fulfilled. Validation: Confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use are fulfilled. Note 1 - In design and development, validation concerns the process of examining a product to determine conformity with user needs. Note 2 - Validation is normally performed on the final product under defined operating conditions. It may be necessary in earlier stages. Note 3 - "Validated" is used to designate the corresponding status. Note 4 - Multiple validations may be carried out if there are different intended uses. [ISO 8402: 1994, 2.18] -9-
ISO/IEC CD 14598-1
v6.4 20 July 1995
5. Overview of software product evaluation standards
5. 1 Structure of standards ISO 9126 defines and gives examples of software quality characteristics and metrics. ISO 14598 gives an overview of software product evaluation processes and provides support for evaluation. Figure 1 shows the relationship between these standards.
ISO /IEC 9126 Software quality characteristics and metrics ISO/ IEC 14598 Software p roduct evaluation
1: Ge neral guide 2: Planning and management 1: Quality characteristics 3: Process for Develop ers 2: External metrics 3: Internal metrics 6: Evaluation modules 4: Process for Acquirers 5: Process for Evaluators
Figure 1 Relationships between standards 5. 2 Evaluation process The ISO/IEC 14598 series of standards provides guidance and requirements for the evaluation process in three different situations: - Development (Enhancement) (ISO/IEC 14598-3) - Acquisition (ISO/IEC 14598-4) - Evaluation (including third-party evaluation) (ISO/IEC 14598-5) 5. 2. 1 Process for developers ISO/IEC 14598-3 should be used by organizations that are planning to develop a new product or enhance an existing product and intending to perform product evaluation using members of its own technical staff. It focuses on the use of those indicators that can predict end product quality by measuring intermediate products developed during the life-cycle. 5. 2. 2 Process for acquirers ISO/IEC 14598-4 should be used by organizations that are planning to acquire (or buy) a software product that will be developed (or has been developed). 5. 2. 3 Process for evaluators ISO/IEC 14598-5 should be used by evaluators carrying out an independent assessment of a software product. This evaluation could be performed at the request of either a developer, acquirer or some other party. This part is intended for those who perform independent evaluation. Often they work for third party organisations. 5. 3 Support for evaluation Each of the evaluation process standards can be used in conjunction with ISO/IEC 14598-2 (Planning and Management) and ISO/IEC 14598-6 (Evaluation modules) standards (Figure 2). -10-
v6.4 20 July 1995
ISO/IEC CD 14598-1
2. Planning and Management
3. Process for Developers
4. Process for Acquirers
Evaluation 5. Process for Process Model Evaluators
6. Evaluation Modules
Figure 2. Relationship of evaluation process to evaluation support standards 5. 3. 1 Planning and Management ISO/IEC 14598-2 Planning and Managing describes planning for software measurement and is applicable to all audiences. When evaluations are to be done, planning is important. This part gives guidance on how to develop a plan for software measurement and provides an example of a plan. 5. 3. 2 Evaluation modules ISO/IEC 14598-6 provides guidance for developing, documenting and validating evaluation modules. These modules represent the specification of the quality model and corresponding metrics that will be applied to a particular evaluation. Evaluation modules are designed and used for each software product evaluation. 5. 4 Software Quality Characteristics and Metrics Each of the above ISO/IEC 14598 standards can be used in conjunction with the ISO/IEC 9126 standards describing Software Quality Characteristics and Metrics: - Quality Characteristics and Subcharacteristics (ISO/IEC 9126-1) - External Metrics (ISO/IEC 9126-2) - Internal Metrics (ISO/IEC 9126-3) Figure 3 shows the relationship between the three standards:
1. Quality Characteristics
2. External Metrics
3. Internal Metrics
Figure 3. ISO/IEC 9126 Software Quality Characteristics and Metrics -11-
ISO/IEC CD 14598-1
v6.4 20 July 1995
5. 4. 1 Quality Characteristics and Subcharacteristics ISO/IEC 9126-1 defines a set of quality characteristic and corresponding subcharacteristics. This standard is used as the foundation for constructing the top two levels of the quality model. The characteristics are expressed in terms which humans use to communicate about the quality of the product being evaluated. On the other hand the subcharacteristics are at an intermediate level between the characteristic (user view) and the metric (quantitative measure of some attribute). These subcharacteristics are usually associated with terms that describe quality with respect to the software products. As such these subcharacteristics become a bridge between the human perspective and the software perspective of quality. 5. 4. 2 External metrics ISO/IEC 9126-2 describes those metrics that represent the external perspective of software quality when the software is in use. Physical values like time and effort are therefore used as the basis for the measures. These measures apply in both the testing and operation phases. When used during test they are meant to be early predictors of the levels of quality that can be expected once the software is used and operated. These measures generally represent the quality in terms that people use to communicate it. 5. 4. 3 Internal metrics Internal produce metrics measures that are used to represent the quality of the static representation of the software. They measure internal requirements of the software related to design and code. These early measures are used to predict what can be expected once the system is in test and operation. Therefore the internal measures are most important to development managers since they are a valuable device for forestalling down stream problems. Internal measures are used to predict the values of corresponding external measures. The internal metrics standard (ISO/IEC 9126-3) shows the relationship between external and internal metrics.
6. Evaluation process
When evaluating software quality, it is necessary to establish the evaluation requirements, and specify, design and execute the evaluation (see Figure 4). Each step is described in more detail in the clauses indicated. This part of ISO 14598 gives an overview of the process. The other parts of ISO 14598 explain how the process is applied in different circumstances.
-12-
v6.4 20 July 1995
ISO/IEC CD 14598-1
Establish Evaluation Requirements
Establish purpose of evaluation (7.1) Identify types of product(s) (7.2) Specify quality model (7.3) 9126-1 Quality Characteristics 9126-2 External Metrics 9126-3 Internal Metrics 14598-6 Evaluation Modules
Select metrics (8.1) Specification of the Evaluation Establish rating levels for metrics (8.2) Establish criteria for assessment (8.3)
Design of the evaluation
Produce evaluation plan (9.1)
Measure characteristics (10.1) Execution of the Evaluation Compare with criteria (10.2) Assess results (10.3)
Figure 4 Evaluation process
7. Establish evaluation requirements
7. 1 Establish the purpose of evaluation 7. 1. 1 General The purpose of software quality evaluation is to directly support both the production and acquisition of software which meets user needs. The ultimate objective is to ensure that the product provides the required quality in use - that it meets the stated and implied needs of the users (including operators, recipients of the results of the software, or maintainers of software). The purpose of evaluation of intermediate product quality may be to: decide on the acceptance of an intermediate product from a subcontractor; decide on the completion of a process and when to send products to the next process; predict or estimate end product quality; collect information on intermediate products in order to control and manage the process. The purpose of evaluation of final product quality may be to: decide on the acceptance of the product; decide when to release the product; compare the product with competitive products; select a product from among alternative products; assess both positive and negative effect of a target entity when it is used. -13-
ISO/IEC CD 14598-1
v6.4 20 July 1995
Software quality can be evaluated within a defined quality structure throughout the product development and acquisition life-cycle processes defined in ISO/IEC 12207. 7. 1. 2 Acquisition When acquiring a custom made software product, the acquirer needs to establish external quality requirements, specify the requirements to the supplier, and evaluate potential purchases against these requirements before acquisition. Quality requirements for a product to be developed are specified to ensure that a product meets the stated and implied needs of the user. When purchasing a software product, evaluation can be used to compare alternative products and to ensure that the selected product meets the quality requirements (see ISO/IEC 14598-4 for the process for acquirers and ISO/IEC 14598-5 for the process for evaluators (including third party evaluation requirements). 7. 1. 3 Development Software product requirements express the users needs for the software product under consideration, and are defined prior to the development. As a software product is decomposed into major components, the requirements derived from the overall product may differ for the different components, and may require different evaluation criteria. Prior to quality evaluation, quality requirements need to be specified in terms of quality characteristics and possible subcharacteristics (see ISO/IEC 9126-1). At the initial stage of evaluation, these quality requirements need to be studied and identified, for planning and implementing evaluation. The developer needs to establish external quality requirements for each relevant quality characteristic. The completeness and correctness of the quality requirements specification needs to be evaluated to ensure that all the necessary requirements have been specified and unnecessary requirements excluded. The developer needs to evaluate the product against these requirements before delivery. It is important to check that implied needs are specified in sufficient detail for all relevant characteristics. If possible, requirements should be assessed by procurers or purchasers, and by end users to assess the implied needs. User experience with prototypes frequently leads to a more accurate statement of requirements for quality in use. The developer should identify internal quality requirements. When internal quality requirements are used, the developer needs to identify these using a quality model which relates them to the external quality requirements, and use the internal requirements to verify quality of intermediate products during development. Software evaluation should be used to predict and verify quality during development, by specifying internal quality requirements for intermediate products in the development process. The external quality of the complete product for specific intended uses can subsequently be evaluated against initial requirements. (See ISO/IEC 14598-3 for the evaluation process for developers). 7. 1. 4 Supply The supplier can use the results of software quality evaluation to obtain feedback on the extent to which different production processes, design methods or CASE tools can be used to meet quality requirements.
-14-
v6.4 20 July 1995
ISO/IEC CD 14598-1
7. 1. 5 Operation The organisation which operates a software system can use software quality evaluation to validate that quality requirements are met under different operating conditions, and to provide feedback on the need for any changes to those responsible for maintenance. 7. 1. 6 Maintenance The organisation which maintains the system can use software evaluation to validate that quality requirements are still met, and requirements for maintainability and portability are achieved. 7. 2 Identify types of product(s) to be evaluated The type of product to be evaluated will depend on the stage in the life cycle and the purpose of the evaluation. The objective is that when the product is actually used by the user it meets the stated and implied needs and thus has quality in use (see Figure 5). External quality can only be assessed for a complete hardware/software system of which the software product is a part. External characteristics are those which are evaluated when executing the software. The values of external measures necessarily depend on more than the software, so the software has to be evaluated as part of a working system. For software to have quality in use it has to meet user needs to carry out particular tasks in particular hardware and software environments. Software which performs satisfactorily in one environment may show quality defects in another environment. External quality characteristics should therefore be evaluated under conditions which emulate as closely as possible the expected conditions of use. External characteristics are measured when the code is complete, though as it may not be possible to emulate the exact conditions of use (eg network environment and user characteristics), external measures are often only indicators of the actual quality in use. If the external quality requirements are not achieved the results of the evaluation can be used as feedback to modify the software characteristics in order to improve the external quality, thus supporting a continual improvement process
-15-
ISO/IEC CD 14598-1
v6.4 20 July 1995
OPERATION
Needs
Quality in use
use and feedback
EXTERNAL external quality requirements validation
system characteristics
External quality
external metrics
INTERNAL
software characteristics
internal quality requirements verification
Internal quality
internal metrics
Figure 5 Quality in the software lifecycle For the purposes of development, internal quality requirements are defined which enable the quality of intermediate products to be verified. Internal characteristics are those which are evaluated by seeing the internal features (e.g. the specification or source code) of the software. Developers and maintainers are normally the only ones interested in the internal characteristics. Modularity and traceability are examples of internal characteristics. Internal measures directly measure internal attributes and may be indicators of other internal characteristics. Achievement of the required internal quality will contribute to meeting the external requirements of the software in use. Internal software quality characteristics can thus be used as indicators to estimate final software quality (see Figure 6). For example, response time is an important measure required to evaluate the usability and the efficiency of the software, but response time cannot be measured during development. In order to evaluate the efficiency of the product during development, path length could be measured based on the intermediate product or specifications. These could be used as indicators which provided rough estimates of response time under certain conditions.
-16-
v6.4 20 July 1995
ISO/IEC CD 14598-1
It is very important that internal software quality attributes are directly related to external quality requirements, so that the quality characteristics of software products under development (both intermediate and end item software products) can be assessed with respect to final system in-use quality needs. Internal metrics are of little value unless there is evidence that they are related to external quality.
measures actual quality in use
measures of actual usage
indicators
external measures of computer system
measures
external characteristics of computer sytem
measures
indicators
internal measures of software
measures
internal characteristics of software
Figure 6 Relationships between measures and characteristics The specific internal attributes which are relevant to final quality will depend on the intended conditions of use - for an interactive product this will depend on the needs of the eventual end user and task. Other issues which will influence the needs for software product quality include whether the product is being purchased or developed, the stage of development, and the hardware, software and network environment in which the product will be used. External measures of a computer system can also be used as indirect measures of internal software quality. Thus the response time of a computer system can be used to measure the efficiency of the software in a particular computing environment. 7. 3 Specify quality model
In order to evaluate software it is necessary to use a quality model which breaks software quality down into different characteristics. ISO/IEC 9126-1 provides a general-purpose model which defines six broad categories of characteristics of the software in use: functionality, reliability, usability, efficiency, maintainability and portability. These can be further broken down into subcharacteristics which have measurable attributes. Internal software product quality attributes are the measurable properties of a software product that influence its ability to satisfy stated and implied needs. One or more attributes can be used to assess a particular software quality characteristic or sub-characteristic (Figure 7).
-17-
ISO/IEC CD 14598-1
v6.4 20 July 1995
attribute sub-characteristic characteristic
Internal
External
Figure 7: Quality characteristics, sub-characteristics and attributes Sufficient internal and external attributes should be identified for each required sub-characteristic. The actual characteristics and sub-characteristics which are relevant in any particular situation will depend on the purpose of the evaluation. The ISO/IEC 9126-1 characteristics and subcharacteristics provide a useful checklist of issues related to quality, but other ways of categorising quality may be more appropriate in particular circumstances. For example, the users view of the quality of a product in use is defined in ISO 9241-11 in terms of the extent to which users of a product can achieve task goals with effectiveness, efficiency and satisfaction in a particular environment. This is measured by attributes of the interaction including the accuracy and completeness with which users achieve goals, and the task time and resources ...