Methodology A for Certication of Modeling and Simulation Applications
OSMAN BALCI Virginia Tech
Certication of modeling and simulation (M&S) applications poses signicant technical challenges for M&S program managers, engineers, and practitioners. Certication is becoming increasingly more important as M&S applications are used more and more for military training, complex system design evaluation, M&S-based acquisition, problem solving, and critical decision making. Certication, a very complex process, involves the measurement and evaluation of hundreds of qualitative and quantitative elements, mandates subject matter expert evaluation, and requires the integration of different evaluations. Planning and managing such measurements and evaluations requires a unifying methodology and should not be performed in an ad hoc manner. This paper presents such a methodology. The methodology consists of the following body of methods, rules, and postulates: (a) employment of subject matter experts, (b) construction of a hierarchy of indicators, (c) relative criticality weighting of indicators using the analytic hierarchy process, (d) using a rule-based expert knowledge base with an object-oriented specication language, (e) assignment of crisp, fuzzy, and nominal scores for the indicators, (f ) aggregation of indicator scores, (g) graphical representation of the indicator scores and weights, (h) hypertext certication report, and (i) interpretation of the results. The methodology can be used for certication of any kind of M&S application either throughout the M&S development life cycle or after the development is completed. Categories and Subject Descriptors: I.6.4 [Simulation and Modeling]: Model Validation and Analysis; D.2.4 [Software Engineering]: Software/Program Vericationvalidation General Terms: Measurement, Verication Additional Key Words and Phrases: accreditation, certication, credibility assessment, evaluation, quality assessment, verication, validation
1. INTRODUCTION A model is a representation or abstraction of something such as an entity, a system or an idea. Simulation is the act of experimenting with or exercising a model or a number of models under diverse objectives including acquisition, analysis, and training. For example, if the analysis objective is to predict the performance
The research and engineering that have led to the development of the certication methodology and the Evaluation EnvironmentTM software tool were funded by the Naval Surface Warfare Center Dahlgren Division at Virginia Tech and Orca Computer, Inc. Authors address: Osman Balci, Department of Computer Science, 660 McBryde Hall, MC 0106, Virginia Tech, Blacksburg, VA 24061; email: [e-mail]; web: http://manta.cs.vt.edu/balci. Permission to make digital/hard copy of part or all of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for prot or commercial advantage, the copyright notice, the title of the publication, and its date appear, and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specic permission and/or a fee. C 2001 ACM 1049-3301/01/1000-0352 $5.00
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001, Pages 352377.
Methodology for Certication of Modeling and Simulation
353
of a complex system design, we experiment with a model or a distributed set of models representing the system design. If the predicted performance is used in making an acquisition decision, the process is called simulation-based acquisition. If the training objective is to teach military commanders how to make decisions under a combat scenario, we exercise a model or a distributed set of models in an interactive manner by using the trainees as part of the simulation. We refer to a specic simulation created for a particular objective as a modeling and simulation (M&S) application (MSA). The U.S. Department of Defense (DoD) is the largest sponsor and user of MSAs in the world. DoD uses many different types of MSAs such as continuous, discrete-event, distributed, hardware-in-the-loop, human-in-the-loop, Monte Carlo, parallel, and synthetic environments, bringing together simulations and real-world systems. The DoD Instruction 5000.61 states that it is the DoD policy that: . . . models and simulations used to support major DoD decision-making organizations and processes . . . shall be accredited for that use by the DoD component sponsoring the application [DoDI 1996, p. 2]. The DoD Instruction 5000.61 denes accreditation as the ofcial certication that a model, simulation, or federation of models and simulations is acceptable for use for a specic purpose [DoDI 1996, p. 9]. On the other hand, the International Organization for Standardization (ISO) denes accreditation and certication as follows: Accreditation is a procedure by which an authoritative body gives formal recognition that a body or person is competent to carry out specic tasks. [Rae, Robert, and Hausen 1995, p. 287] Certication is a procedure by which a third party gives written assurance that a product, process or service conforms to specied characteristics. [Rae, Robert, and Hausen 1995, p. 288] The above ISO denitions conict with the denitions given in [DoDI 1996]. We use the ISO terminology in this paper. It should be noted that the DoDs denition of accreditation corresponds to the ISOs denition of certication. Component-based development is becoming increasingly important [Brown 1996]. Component-based software development technology creation is currently led by NIST under the advanced technology program on Component-Based Software [NIST 2002]. NIST cites many advantages of component-based development that can also be realized for MSA development conditioned on the following: 1. Establishment of a marketplace for component-based MSA development so that M&S technology users can realize signicant economic benets through (a) reduced M&S project costs, (b) enhanced M&S credibility, and (c) expanded applicability of less expensive technology. 2. Increased automation and productivity in MSA development enabling (a) improved M&S quality characteristics, (b) reduced time to develop, test, and certify MSAs, and (c) increased amortization of costs through M&S component reuse.
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
354
O. Balci
Fig. 1. Accreditation and certication practice.
3. Increased productivity of M&S project teams by (a) permitting specialists in the application domain to create higher delity M&S components, and (b) providing a focus on discourse in M&S development at a level far more comfortable to application domain users than a programming language. 4. Expanded markets for the producers of MSAs and M&S components by promoting (a) the creation of systematically reusable M&S components, (b) increased interoperability among M&S software and non-M&S software products, and (c) convenient and ready adaptation of M&S components. Component-based development is an effective and affordable way of creating MSAs and conducting MSA verication, validation, and certication (VV&C) [Balci et al. 2002; Glasow and Pace 1999]. A veried and validated M&S component can substantially decrease the MSA VV&C effort when reused. Such practice can signicantly decrease the time and cost of MSA development. Component-based MSA development technology might be the silver bullet for effective and affordable MSA VV&C. Similar to the manner accreditation and certication are carried out in other disciplines, Balci et al. [2002] propose the scheme shown in Figure 1. The scheme assumes two scenarios: (a) MSA development under contract, and (b) a marketplace where developers fabricate reusable M&S components for sale, and MSAs are developed by way of component reuse. Under scenario (a), an MSA sponsor hires an independent MSA certication agent, which is accredited by an external accreditation authority to conduct the required certication. Currently, this scenario is practiced in DoD under the label of accreditation, where no authority exists to accredit the work of the agent. Certainly, the MSA sponsor should be assured that the practice of the certication agent satises minimum standards established by an independent accreditation authorityyet to be founded in the United States. Under scenario (b), an M&S component developer fabricates a reusable M&S component for sale. The buyer of such a component seeks assurance about the
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
Methodology for Certication of Modeling and Simulation
355
component quality. An independent M&S component certication body can provide such an assurance by awarding a mark of conformity (e.g., certied to be HLA compliant), a seal of approval, or a certicate of excellence. There may be many certication bodies, which should provide certications that are unbiased, fair, cost-effective, and reproducible. Therefore, the certication bodies should be accredited by an external accreditation authority. For example, the SGS International Certication Services (SGS-ICS) group of companies (http://www.sgsgroup.com/sgsgroup.nsf/pages/home.html) is a leading international certication body, which is accredited in the United Kingdom by the National Accreditation Council for Certication Bodies (NACCB) and in Belgium by the Dutch Council for Certication (RvC). In the United States, for example, the Accreditation Board for Engineering and Technology (ABET) accredits educational programs that award diplomas and certicates. ABET [2002] states that the diversity of educational programs in the United States is one strength of the American educational system. Such a large selection of educational offerings makes quality a vital issue. Accreditation is the quality assurance that education is meeting minimum standards. When the component-based MSA development technology is commonly used and an M&S component marketplace is established, organizations such as SGS-ICS can be founded to independently perform M&S component certication. Organizations similar to NACCB, RvC, and ABET can be founded to accredit, regulate, and monitor the M&S component certication bodies. Certication of software products by independent evaluation has been practiced in the software industry since early 1990s, especially in Europe and recently in the United States. The following references describe how certication of software products and components is currently practiced: [ISACC 1999; Loesh et al. 1999; Poore, Mills, and Mutchler 1993; Rae, Robert, and Hausen 1995; Rodrguez-Dapena 1999; Vermesan 1997, 1998; Voas 1998a,b,c, 1999a,b,c,d, 2000a,b; Wakid et al. 1999; Wallace 1999; Wohlin and Regnell 1998; Wohlin and Runeson 1994]. Certication of M&S products by independent evaluation has been practiced within DoD under the name of M&S accreditation. Certication of M&S products is a very complex process and poses signicant technical challenges for M&S program managers, engineers, and practitioners [Balci et al. 2002]. It involves the measurement and evaluation of hundreds of qualitative and quantitative elements, mandates subject matter expert (SME) evaluation, and requires the integration of different evaluations. Planning and managing such measurements and evaluations require a unifying methodology and should not be performed in an ad hoc manner [Balci et al. 2002]. This paper presents such a methodology. The methods, rules, and postulates that make up the methodology are described in section 2. Section 3 presents a strategy for creating indicators for MSA acceptability assessment for certication and introduces the higher levels of a generic hierarchy of indicators that can be used under the methodology for certication of any kind of MSA. Section 4 provides guidance in applying the methodology based on the authors experience. Conclusions are given in Section 5.
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
356
O. Balci
2. THE METHODOLOGY Preliminary research for the methodology described herein was conducted by the author between 1983 and 1991. Between 1992 and 1995, an earlier version of the methodology was developed for application to complex system design evaluation at Virginia Polytechnic Institute and State University (Virginia Tech) under funding from the Naval Surface Warfare Center Dahlgren Division (NSWCDD) [Talbert 1995]. A software tool, Evaluation EnvironmentTM [Orca Computer 1999a,b], was designed and developed at Orca Computer, Inc. with signicant improvements to the methodology under funding from NSWCDD between 1997 and 1999. The software tool facilitates the application of the methodology. The methodology and the tool have been used since 1999 in many evaluation projects under the U.S. National Missile Defense program. The Merriam-Webster dictionary denes methodology as a body of methods, rules, and postulates employed by a discipline: a particular procedure or set of procedures. Our methodology consists of the following body of methods, rules, and postulates: employment of subject matter experts, construction of a hierarchy of indicators, relative criticality weighting of indicators using the analytic hierarchy process, using a rule-based expert knowledge base with an object-oriented specication language, assignment of crisp, fuzzy, and nominal scores for the indicators, aggregation of indicator scores, graphical representation of the indicator scores and weights, hypertext certication report, and interpretation of the results. Each part of the methodology is described in a separate section below. 2.1 Employment of Subject Matter Experts Subject matter experts (SMEs) are commonly employed for M&S evaluation for certication [Glasow 1998; Pace 1998]. Under the methodology, the knowledge and experience of SMEs are utilized for: constructing a hierarchy of indicators, relative criticality weighting of indicators, building a rule-based expert knowledge base, and assigning scores for the indicators. SMEs should be employed to cover all areas of the problem domain and all phases of the development life cycle. For example, for a ground-based radar (GBR) simulation, technical SMEs knowledgeable about the GBR systems and operational SMEs and veterans who have intimate knowledge and experience about the operations of the GBR systems should be employed. In addition,
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
Methodology for Certication of Modeling and Simulation
357
Fig. 2. Selecting and weighting SMEs qualied to score on a leaf indicator.
technical SMEs knowledgeable about MSA development should be employed. A different set of SMEs may be selected for a different phase of the development life cycle, such as requirements, design, implementation, and integration. Usually an SME is qualied to assess some of the leaf indicators in the hierarchy. For example, a technical SME specializing in M&S designs may be designated to assess only those leaf indicators dealing with the M&S design assessment. Therefore, SMEs must be identied to score on a particular leaf indicator based on their expertise, as depicted in Figure 2. The SMEs who are qualied to assess a leaf indicator may have different levels of expertise. One SME may be more knowledgeable than another; therefore, his or her assessment should be taken into consideration more than the others. Thus, the qualied SMEs must be weighted among themselves. Figure 2 depicts the relative criticality weighting of the three qualied SMEs using the Eigenvalue method underlying the analytic hierarchy process discussed in section 2.3. 2.2 Construction of a Hierarchy of Indicators Measurement and evaluation of qualitative and quantitative concepts is an area of interest in many disciplines. Examination of these disciplines reveals a common approach that uses, what we call, indicators. An indicator is an indirect measure of a qualitative concept (e.g., M&S design quality) or a direct measure of a quantitative concept (e.g., utilization). The following terms are used in the designated disciplines to imply what we mean by indicators: metrics (software engineering); measures, indexes (computer performance evaluation); measures of effectiveness, measures of merits, measures of performance (engineering);
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
358
O. Balci
Fig. 3. A hierarchy of indicators.
and indicators (economics, psychometrics). The term indicator is used in this paper to be synonymous to any one of the terms listed above. Figure 3 shows how a qualitative concept can be measured by using a hierarchy of indicators. Since the qualitative concept cannot be directly measured, a set of indicators is created to measure it indirectly at the top level. However, the top-level indicators may still not be directly measurable and therefore, a set of indicators is created to measure each one of them. Those indicators that are not directly measurable in the new set are further decomposed into other indicators. Decomposition of indicators continues until the leaf indicators (i.e., the ones that are not further decomposed) can be directly measured or assessed. This decomposition results in a hierarchy of indicators with the structure depicted in Figure 3. Note that the structure is not a tree. An indicator may inuence more than one parent indicator. Three types of indicators exist in the hierarchy: root, branch, and leaf. 1. The root indicator is at the apex of the hierarchy representing the qualitative concept. For MSA certication, the root indicator is the MSA acceptability that must be measured and evaluated to formulate a certication decision. 2. A branch indicator is one that has at least one parent indicator and at least one child indicator. 3. A leaf indicator is one that has at least one parent indicator and no child indicator. Only the leaf indicators need to be assessed in the hierarchy. The assessments of leaf indicators are aggregated throughout the hierarchy in a bottom-up fashion as discussed in section 2.6. A certication decision for an MSA is made based on the results of the acceptability assessment of the MSA. MSA acceptability is a qualitative concept that cannot be directly measured or assessed. Therefore, indicators are created to indirectly measure it. The following indicators are proposed at the top level: MSA Formulated Problem Credibility MSA Requirements Credibility
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
Methodology for Certication of Modeling and Simulation
359
MSA Application Credibility MSA Experimentations Credibility MSA Project Management Quality MSA Cost MSA Risk However, none of these indicators can be directly measured or assessed, requiring further decomposition of each into a set of other indicators. The higher levels of a generic hierarchy of indicators for assessing the acceptability of any kind of MSA are presented in section 3. 2.3 Relative Criticality Weighting of Indicators Using AHP As depicted in Figure 3, a child indicator may have more than one parent and it inuences the scores of all of its parent indicators. For example, the model complexity indicator may inuence the scores of its parent indicators: model development cost, model maintainability, model reusability, model reliability, and model testability. A child indicator may inuence the score of a parent indicator differently from the other sibling indicators. For example, the effect of model complexity on model reusability may be much higher than its effect on model development cost. Therefore, the child indicators must be weighted among themselves in their inuence on the score of a parent indicator. Since a child indicator can inuence more than one parent indicator, it can have more than one weight. Weights are used to express a child indicators level of inuence. A relative weight is a fractional value between zero and one. The weights of the child indicators belonging to the same parent must sum to one. Since the weights are assigned relative to the sibling indicators to express criticality of inuence on the parent indicator, the process is called relative criticality weighting. Given a list of n indicators, it is very difcult for an SME to come up with weights especially when n > 5. To facilitate the relative criticality weighting among n indicators or decision elements, the mathematical approach called the analytic hierarchy process (AHP) is often used in the multicriteria decision making eld [Saaty 1980, 1990, 1994]. AHP enables the SME to make pairwise comparisons of importance between sibling indicators and computes the weights based on the SMEs pairwise comparisons using methods such as the eigenvalue, mean transformation, and row geometric mean methods. Saatys pairwise comparison scale is described in Table I. This scale is considered to be the standard in the published literature. The use of the scale and the pairwise comparisons are illustrated in Figure 4. The sibling indicators to be weighted are presented in the form of a matrix as shown in Figure 4. Originally, each matrix cell has a value of 1 indicating equal weighting. The SME clicks on a matrix cell to display the names of the two corresponding indicators for comparison. Figure 4 illustrates the comparison of MSA Requirements Traceability versus MSA Requirements Modiability. If the relationship is the reverse, i.e., modiability is more important than
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
360
O. Balci
Table I. Pairwise Comparison Scale
Comparative Importance 1 3 5 7 9 2, 4, 6, 8
Denition Equally important Moderately more important Strongly more important Very strongly more important Extremely more important Intermediate judgment values
Reciprocals
Explanation Two sibling indicators equally inuence the parent indicators score. One sibling indicator is moderately more inuential than the other. One sibling indicator has stronger inuence than the other. One sibling indicator has signicantly more inuence over the other. The difference between inuences of the two indicators is extremely signicant. Judgment values between equally, moderately, strongly, very strongly, and extremely. If v is the judgment value when i is compared to j , then 1 /v is the judgment value when j is compared to i.
traceability, then the corresponding cell on the other half of the matrix is selected. A judgment is specied by clicking on the sliding bar scale or by sliding the knob on the scale. The reciprocal of the judgment value is given to the corresponding cell on the other half of the matrix. Once the SME has performed all pairwise comparisons, a method (e.g., eigenvalue method) is selected from the pop-up menu and the OK button is clicked, whereupon the fractional weights are computed by using the AHP technique. SMEs may not be consistent in their judgments [Saaty 1990]. Before computing the weights, the degree of inconsistency is measured by the Inconsistency Index (II). Perfect consistency implies a value of zero for II. However, perfect consistency cannot be demanded since, as human beings, we are often biased and inconsistent in our subjective judgments. Therefore, the SME is not alerted about the inconsistency if II 0.1. For II values greater than 0.1, the SME is alerted and given a chance to revise his or her judgments. 2.4 Using a Rule-Based Expert Knowledge Base The MSA acceptability should be assessed to make a certication decision by incorporating knowledge about the problem domain for which the MSA is intended, relationships and dependencies among the indicators, technical characteristics of the MSA, and the requirements under which the MSA is developed. An object-oriented language was developed and implemented for rule-based knowledge specication [Orca Computer 1999a,b]. Using this language, rules can be constructed to provide four types of action: 1. Prompters are actions that require the SME to enter an explanation, justication, or description during the evaluation.
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
Methodology for Certication of Modeling and Simulation
361
Fig. 4. Pairwise comparisons of indicators under AHP.
2. Triggers are internal actions that are not visible to the SME; they cause the occurrence of internal actions such as the recording of some values, comparisons, computations, and knowledge-based inferencing. 3. Alerters are actions that notify the SME about undesirable conditions. 4. Informers are actions that provide useful information to the SME during the evaluation. A set of rules can be constructed to guard against inconsistencies and biases that may be exhibited by SMEs during the evaluation. Other rules can be created and executed manually to query the information contained within a hierarchy of indicators. 2.4.1 Class Library. A class library is provided to enable the user to access the built-in classes shown in Figure 5. This Figure also shows the class
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
362
O. Balci
Object String Iterator Range ObjectWithValues Expert Indicator Project Console
Fig. 5. Class inheritance hierarchy.
Table II. Message Passing Forms Message Passing Forms [object message] Can be Used as a Statement? Yes Can be Used in an Expression? Yes
[message of object] [message] tell object to message tell object that message
Examples As a statement: [console print:[sme name]]; In an expression: set rootIndicator to [project topIndicator]; if [[expertNamed: Albert Einstein of project] hasCompletedEvaluation] then . . . set childIndicators to [children]; tell console to println: The score range = & [aggregateScore of [childNamed: Risk]]; if [[aggregateScore] low] < 60 then tell console that warning: Score is too low;
No No Yes Yes
Yes Yes No No
inheritance structure. A class has its own characteristics (i.e., instance variables, class variables) and behavior (i.e., instance methods, class methods), and inherits other characteristics and behavior from its ancestor classes. For example, the class Expert inherits the characteristics and behavior of its parent class ObjectWithValues and grandparent class Object. When an object (e.g., an SME, an indicator, a project, an iterator) is instantiated from its corresponding class, it inherits all of the characteristics and behavior of its class. This implies that the instantiated object can respond to all messages (i.e., method invocations) provided by its class. Altogether, 81 methods are available for use in the class library. 2.4.2 Message Passing. All objects instantiated from the various classes communicate with each other via message passing. Each object has a unique object reference that is used for sending a message to that object. Table II shows the message passing forms that can be used as statements or in expressions. The brackets imply message passing and can be embedded. For example, [console print:[sme name]] indicates that the message called name is sent to the object pointed to by the object reference sme. This message passing returns the name of the subject matter expert, which becomes the value of the
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
Methodology for Certication of Modeling and Simulation
363
parameter of the method called print. The colon indicates a parameter specication for a method. Then the print message with the parameter value just returned, is sent to the console object, which prints the name to the console window. If no object reference is specied in a message passing, for example, [children], the object pointed to by the default object reference, self receives the message. Self is an object reference pointing to the indicator object for which the rule is written. Thus, the message passing [children] is equivalent to [children of self] or [self children]. A message can be sent to any indicator in the hierarchy. If the object reference of the indicator of interest is stored in a variable, the variable name, for example, rootIndicator, is used for sending the message; otherwise, the object reference can be inserted in a rule by drag-and-drop. After clicking on the insertion point in a rule, while holding down the Alt key, the name of the indicator of interest in the browser is clicked, dragged, and dropped at the insertion point. A small icon is inserted representing the object reference of the indicator. 2.4.3 Rule Specication. A rule is specied in the form of a condition and action pairspecically for a particular indicator. For example, three rules are shown in Figure 6 for the indicator MSA Requirements Consistency. The left box is used for condition specication, the right box is used for action specication, and the lower box is used for rule documentation. The rules can be ordered by using the up and down arrow buttons. A condition is specied as a method that must return true or false upon execution. An action is also specied as a method but it does not return true or false; it simply performs a set of actions. When the SME assigns a score to or changes the weight of an indicator, the conditions of the rules specied for that indicator are scanned from top to bottom. If a condition method returns true, the corresponding action method is executed. Figure 7 shows a more complex action method that veries the consistency of indicator weights in the entire indicator hierarchy. This action method can be specied with any indicator in the hierarchy with a condition method return true;. The rule is executed manually. 2.5 Assignment of Crisp, Fuzzy, and Nominal Scores for the Indicators For a leaf indicator, an SME can assign a crisp, fuzzy or nominal score, dened as follows. 1. A crisp score is a single real value between 0 and 100 (e.g., 75.8). 2. A fuzzy score is an interval of real values within the range of 0 and 100 (e.g., [72.5, 83.7]). The end points of the interval are always inclusive. 3. A nominal score is a named score with a predened crisp or fuzzy value. Table III shows an example set of nominal scores with predened numerical fuzzy values or score ranges. Note that the high and low scores in adjacent score ranges are the same, since the score values are real. The plus sign means
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
364
O. Balci
Fig. 6. Example rules.
more and implies an increase in the goodness or badness of the assessment. The minus sign means less and implies a decrease. The plus sign goes upward for positive assessments, and downward for negative assessments; for example, Poor + means more poor and Poor means less poor. Therefore, Poor + has a lower score range than Poor . Many approaches exist for determining a score for a leaf indicator. Four approaches are described below in no particular order. 2.5.1 Testing. Two types of testing can be performed: software testing and eld testing. In software testing, tests are designed to perform either verication or validation or both. More than 100 techniques are available for MSA software
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
Methodology for Certication of Modeling and Simulation
Iterator ref Indicator ref Real descendants, children; indicator, childIndicator; sumOfWeights;
365
set indicator to [project topIndicator]; set descendants to [indicator descendants]; repeat while indicator != nil do { if [numberOfChildren of indicator] > 0 then { tell console to println:"\"" & [indicator name] & "\""; set children to [indicator children]; set childIndicator to [nextObject of children]; set sumOfWeights to 0.0; repeat while childIndicator != nil do { tell console to println:"\t\"" & [childIndicator name] & "\"s weight = " & [childIndicator defaultWeightForParent:indicator]; add [childIndicator defaultWeightForParent:indicator] to sumOfWeights; set childIndicator to [nextObject of children]; } if (1 - sumOfWeights) > 0.00001 or (sumOfWeights - 1) > 0.00001 then tell self to emitError:"The weights of the child indicators of \"" & [indicator name] & "\" do not sum to 1.\n"; // emitError: message stops execution of the remaining statements. } set indicator to [nextObject of descendants]; } tell console to println:"\n*** Congratulations! All indicator weights are found to be consistent. ***\n";
Fig. 7. An action method verifying the consistency of indicator weights.
Table III. Example Nominal Nominal Scores Score Excellent + Excellent Excellent Good + Good Good Satisfactory + Satisfactory Satisfactory Poor Poor Poor + Unacceptable Unacceptable Unacceptable + Numerical Score [97 .. 100] [94 .. 97] [90 .. 94] [87 .. 90] [84 .. 87] [80 .. 84] [77 .. 80] [74 .. 77] [70 .. 74] [67 .. 70] [64 .. 67] [60 .. 64] [40 .. 60] [20 .. 40] [0 .. 20]
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
366
O. Balci
testing [Balci 1998; Binder 2000]. The test results are transformed into value judgments, which determine scores assigned to the leaf indicators. Field testing places the MSA in an operational situation (e.g., actual military exercises) for the purpose of collecting as much information as possible for testing the MSA. For example, for acceptability assessment of an MSA being developed for the National Missile Defense system design evaluation, actual eld tests such as integrated ight tests (IFTs), integrated ground tests (IGTs), and integrated system tests (ISTs) were performed. The data collected during the IFTs, IGTs, and ISTs were used in assessing many leaf indicators. 2.5.2 Direct Measurement. Certain elements of an MSA can be directly measured, once the executable version is created. For example, for a distributed interactive MSA, the following elements can be directly measured: animation speed, network bandwidth, execution time, amount of computer memory used, and amount of disk space used. The measured value can be a crisp value or a statistical condence interval (fuzzy value). An SME can judge the goodness of the value and assign a crisp, fuzzy or nominal score to express his or her judgment for the indicator. If the system represented by the MSA exists and data can be collected about its input-output behavior, direct measurement of the system can be used. The data collected by instrumenting (i.e., inserting probes, breakpoints, and traps into) the real system are analyzed to make value judgments based on which scores are assigned to the leaf indicators. 2.5.3 Analysis. The analysis can be carried out by using a variety of techniques such as mathematical analysis, statistical analysis, probabilistic analysis, and metrics analysis. For example, the analysis results obtained by using a queuing theory model (e.g., M/M/s) can be used in judging the validity of an MSA component. The analysis results are interpreted by SMEs and converted into value judgments based on which scores are assigned to the leaf indicators. Metrics are commonly used in the eld of software engineering. A metric is a formula computed and evaluated to measure a certain aspect of the entity of interest (e.g., model complexity, model cohesion, model coupling). Metrics are commonly used for comparative analysis and metric values must be interpreted. An SME can interpret a metric value and assign a score to a leaf indicator. 2.5.4 Examination. An SME, based on his or her technical M&S expertise, can examine the MSA and assign a score for a leaf indicator. The SME is assumed to have expertise and experience about a particular phase, several phases, or all phases of the MSA development life cycle. For example, an SME on MSA implementation with years of experience can judge the goodness of an MSA implementation. Certain indicators can only be assessed by SMEs who possess domain specic operational knowledge. For example, certain quality indicators of a GBR simulation can only be assessed by veterans who possess operational experience with the GBR systems. Such SMEs, although difcult to nd, can be asked to examine the MSA and assign scores to the leaf indicators.
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
Methodology for Certication of Modeling and Simulation
367
Fig. 8. An example Kiviat graph showing fuzzy scores for the sibling indicators.
2.6 Aggregation of Indicator Scores Only the leaf indicators are scored on. The scores of the leaf indicators are aggregated in a bottom-up fashion throughout the indicator hierarchy. The aggregation is carried out by the weighted average of scores with respect to the weights of child indicators and with respect to the weights of SMEs. 2.7 Graphical Representation of the Indicator Scores and Weights Variations of Kiviat graphs are used to graphically represent the indicator scores and weights, as depicted in Figure 8. Each radius of the circle represents an indicator with its name shown just outside of the circumference. The center of the circle represents a score of zero and the circumference intersection point represents a perfect score of 100. The indicator score is designated on the radius either as a crisp value or an interval. The score closest to the circumference is the high score, the score closest to the center of the circle is the low score, and the middle score is the average score. Kiviat graphs enable the joint visual assessment of many sibling indicators. The graphical shape formed by connecting the indicator scores represents an overall assessment. The shape visually displays the high, average, and low scores of all indicators. By examining the form of the shape, one can visually perceive how well the assessment is with respect to each indicator. A perfect,
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
368
O. Balci
ideal Kiviat graph has the shape of a circle, where each indicator has a perfect score of 100. 2.8 Hypertext Certication Report The Evaluation EnvironmentTM software tool [Orca Computer 1999a] can generate a hypertext certication report in the HyperText Markup Language (HTML) and Rich Text Format (RTF). The report includes Certication project documentation Information about the SMEs Hierarchical list of indicators Alphabetical list of indicators Leaf indicators report Kiviat graphs of Aggregate scores for an indicators children Default weights for an indicators children Scores assigned by the SMEs for an indicator Weights of the SMEs for an indicator The report can instantly be published on the World Wide Web (Web) and viewed by all people involved in the MSA project using a Web browser. When publishing the certication report on the Web, the identity of each SME can be hidden. 2.9 Interpretation of the Results MSA acceptability assessment results are descriptive in nature, and they need to be interpreted to reach a certication decision. Often a certication decision is a condence building activity. As such, the MSA acceptability root indicator score should be considered as one of the factors affecting the MSA certication decision. Other factors should also be considered in reaching a nal MSA certication decision. Example factors include: independence in assessment, quality of the hierarchy of indicators, quality of SMEs, the scores assigned by SMEs, aggregated scores throughout the hierarchy, weights, and SME comments. Presentation of indicator scores as interval values enables worst-case (pessimistic) analysis, most-likely analysis, and best-case (optimistic) analysis. Using the low-end values of the interval scores, the MSA acceptability can be judged under the worst-case scenario. Use of the mid-point values in the analysis provides a most likely assessment of acceptability. Using the high-end values of the interval scores, the acceptability can be assessed under the best-case scenario. 3. A HIERARCHY OF INDICATORS Two of the strategic directions in verication, validation, and accreditation (VV&A) research and practice advocated by Balci et al. (2002) are: VV&A should be expanded from accuracy-centered assessment to qualitycentered assessment.
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
Methodology for Certication of Modeling and Simulation
369
Fig. 9. MSA development life cycle products and processes.
VV&A should be expanded from product-centered assessment to (product/ process/project)-centered assessment. The IEEE Standard 1059 indicates that Software verication and validation employs review, analysis, and testing techniques to determine whether a software system and its intermediate products comply with requirements. These requirements include both functional capabilities and quality attributes [IEEE 1993, p. 4]. The IEEE Standard 1059 includes quality assessment within the verication and validation (V&V) activities by listing 19 quality attributes (also called quality characteristics) including efciency, interoperability, maintainability, reliability, reusability, testability, and usability. A generic MSA development life cycle consists of processes and (work) products as shown in Figure 9. Process refers to a series of activities conducted to create a life-cycle product, such as engineering the MSA requirements, designing the MSA, or creating the MSA executables (implementation). Product refers to a different representation of the MSA during its development life cycle, such as the MSA requirements specication, MSA design specication, or MSA code (executables). An MSA representation is transformed from one product form (e.g., design specication) into another (e.g., executables) by carrying out a process (e.g., implementation) during the development life cycle. The MSA acceptability assessment for certication should be concerned not only with the assessment of product quality, but also with the quality of the process used to create the product. The quality of the MSA project management also affects the overall MSA acceptability. Certainly documentation plays a critical role in the acceptability assessment, and documentation quality should also be considered. Consequently, the strategy for creating a generic hierarchy of indicators for MSA acceptability assessment for certication dictates the assessment of (a) quality of the product, (b) quality of the process used to create the product, (c) quality of the MSA project management, and (d) quality of the documentation that describes the product, process, and quality assurance of the product and process. The strategy is illustrated in Table IV.
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
370
O. Balci
Table IV. Strategy for Creating Indicators for MSA Acceptability Assessment for Certication Credibility Product Quality Accuracy Quality Characteristic 2 Quality Characteristic 3 : Quality Characteristic k Quality of Approach Used : Quality of Human Resource Management : Product Documentation Quality Process Documentation Quality Quality Assurance Documentation Quality Verity Validity
... ... ... ... ... ...
Process Quality Project Quality Documentation Quality
Table IV implies that the MSA acceptability is inuenced by the credibility of a phase of the MSA development life cycle. The credibility is inuenced by the qualities of the product, process, project, and documentation of that phase. Product quality is the degree to which the product possesses a desired set of characteristics. The rst product quality characteristic product accuracy is assessed by evaluating product verity and validity. Product verity is evaluated by conducting product verication and product validity is evaluated by conducting product validation. Product verication deals with the transformational accuracy of that product and addresses the question of Are we building the product right? Product validation deals with the representational or behavioral accuracy of that product and addresses the question of Are we building the right product? We refer to product verication and product validation as simply V&V throughout the MSA development life cycle. Other product quality characteristics change from one MSA project to another and are determined based on the MSA requirements and intended uses [Balci and Ormsby 2000]. Process quality is the degree to which the process possesses a desired set of characteristics. The set of desired characteristics depends on the process methodologies and techniques employed by the MSA developer. CMU SEI [1994] has developed the Software Capability Maturity Model (SW-CMM) as an application of the process management concepts of total quality management to software. SW-CMM is now very commonly used in the software industry as a means of judging software development process maturity and quality. Many indicators can be developed based on SW-CMM and applied to the assessment of process quality. Project quality is the degree to which the project possesses a desired set of characteristics. Project quality is assessed by evaluating characteristics such as cost management, human resource management, integration management, quality management, risk management, and time management.
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
Methodology for Certication of Modeling and Simulation
371
Document (or documentation or report) quality is the degree to which the document possesses a desired set of characteristics. The quality of a document is mostly determined by the quality of its content; however, other quality characteristics are also important. We have developed a hierarchy of more than 80 indicators to assess the document quality other than the quality of the documents content, which should be assessed in other V&V activities. The top-level indicators of this hierarchy are given as: accessibility, accuracy, completeness, consistency, clarity, maintainability, portability, and readability. Following the strategy outlined in Table IV, a hierarchy of indicators can be developed for MSA acceptability assessment for certication. At the top level of the hierarchy, the following indicators can be dened to inuence the MSA acceptability: MSA Formulated Problem Credibility MSA Requirements Credibility MSA Application Credibility MSA Experimentations Credibility MSA Project Management Quality MSA Cost MSA Risk The acceptability is affected by the credibility of the formulated problem, which is expected to be solved by the creation and use of the MSA. An MSA is created to represent a system dened with respect to the formulated problem. If the formulated problem does not entirely contain the real problem, type III error occurs. Type III error is the error of solving the wrong problem [Balci and Nance 1985]. Based on the work of Balci and Nance [1985], we have developed a hierarchy of more than 85 indicators to assess the credibility of a given formulated problem. Credibility assessment of MSA requirements is especially critical for MSAs created for training purposes. This indicator addresses the question: Will the MSA meet the real needs of the application sponsor if it is developed under the specied requirements? An MSA that does not meet the real needs for which it is created cannot be certied. Credibility assessment of an MSA deals with the assessment of how well the application is developed throughout its entire development life cycle. It includes the assessment of conceptual model credibility, design credibility, implementation credibility, integration credibility, data credibility, conguration management quality, overall product quality, and documentation quality. Designing and conducting statistical experiments with an MSA are known to be difcult. An MSA may be developed to be perfectly acceptable; but, if the simulation experiments are not properly designed and conducted under the prescribed intended uses [Balci and Ormsby 2000], the simulation results become erroneous. Therefore, the credibility of experimentations with an MSA signicantly inuences the MSA acceptability.
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
372
O. Balci
Quality of MSA project management affects our condence in the MSA acceptability. Many management quality characteristics inuence our condence, including the quality of project management of: planning, integration, scope, time, cost, quality, human resource, communications, risk, and procurement. Quality of the certication (or VV&A) plan and the quality of its execution also affect our condence in the MSA acceptability [Balci et al. 2000]. Certainly, cost is a factor for acceptability. For example, an MSA created for training military personnel may not be acceptable if the cost of its use (e.g., performing training exercises), cost of its maintenance, cost of its technical support, and cost of learning how to use it, is beyond the budget of the application sponsor or potential user. Any MSA has risks associated with its development. An MSA may not be able to pass the acceptance criteria specied in the contract, it may not satisfy the performance requirements, or different components developed by subcontractors may not be able to be integrated. Such risks should be assessed periodically during the development life cycle and should be minimized as much as possible to increase the probability of acceptance. 4. APPLICATION OF THE METHODOLOGY There are three major approaches with which the methodology can be applied: 1. The MSA sponsor hires a certication agent before the contractor starts the development and requires the agent to apply the methodology throughout the entire MSA development life cycle to assess the MSA acceptability for a given set of intended uses. 2. The sponsor or potential user of an MSA hires a certication agent after the contractor completes the development of a new MSA or after a contractor modies an existing MSA, and tasks the agent to assess the MSA acceptability for a given set of intended uses by applying the methodology. 3. The sponsor or potential user of an already developed or legacy MSA hires a certication agent to assess the MSA acceptability for a given set of intended uses by applying the methodology. Approach 1 provides many advantages, including the following: The MSA developer gets feedback for acceptability as the MSA development progresses throughout the life cycle. MSA errors and deciencies are discovered early in the development life cycle, resulting in signicant cost savings. The complexity of MSA acceptability assessment is signicantly reduced. The Probability of Type II Error (MSA Users Risk) is signicantly reduced. Type II Error is the error of certifying an MSA when in fact it should not have been certied. Communication between the independent certication agent and the MSA developer helps improve the MSA quality.
ACM Transactions on Modeling and Computer Simulation, Vol. 11, No. 4, October 2001.
Methodology for Certication of Modeling and Simulation
373
The MSA developer is encouraged to implement an effective software quality assurance program. MSA product and resource risks are signicantly reduced. General step-by-step instructions and guidelines for a certication agent to follow when applying the methodology are provided below. The steps may be revised, depending on which of the approaches is used. Step 1. Employ SMEs who have technical and operational knowledge about the problem/application domain and about the development of MSAs in that domain. Step 2. Working with SMEs, create a hierarchy of indicators specically for the MSA, either as a whole or in part or for a particular phase of its development life cycle (e.g., requirements, design, implementation) or for a particular MSA build (increment or version) during its development. Note that acceptability, credibility, or quality is always assessed with respect to a set of intended uses [Balci and Ormsby 2000]. Different hierarchies of indicators may be required for different sets of intended uses. Section 3 provides some guidance for creating indicators. Step 3. Form an independent review panel, and have the panel assess the appropriateness of the hierarchy of indicators created. The panel may consist of SMEs, veterans, engineers, managers, and ot...