of Testing
Table Contents
A13. Testing 1
Levels of Verification
The Unreachable Goal: Correctness
A13. Testing 2
Levels of Verification Testing and Errors Life Cycle Testing Integration Testing System Testing Function Testing Acceptance Testing Testing Experiment Exhaustive Testing Testing Principles Testing Mechanics White Box Testing White Box: Logic Testing White Box: Path Testing Test Path Determination Path Input Domains Reverse Execution Reverse Path Test Example Reverse Path Test Example (cont) Testing Reliability Mutation Analysis Mutation Analysis Process Error Seeding Error Seeding Process
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD Computer Science Dept Va Tech Aug., 2001
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Testing and Errors
A13. Testing 3
Life Cycle Testing
Testing Phases Requirements Specification High Level Design Low Level Design Acceptance Test
A13. Testing 4
Relationship between Discovered Errors and Undiscovered Errors Probability of Existence of More Errors
Function Test System Test
Probability Increases as Number of Errors Increases Number of Errors Found to Date
Coding Integration Test Unit Test Integration Testing Deployment Regression Test
40-50% of all development time is spent in the testing process Humans (programmers) are NOT good at testing. The process of testing admits that one has produced code with errors. Successful testing can be thought of as successfully finding errors and testing failure implies not discovering any errors.
Maintanence
"Testing can establish the presence of errors, but never their absence." "Testing can establish the presence of errors, but never their absence." [Edsger Dijkstra] [Edsger Dijkstra]
Regression Testing involves fixing errors during testing and the reexecution of all previous passed tests. Unit Testing utilizes module testing techniques (white-box / blackbox techniques). Integration Testing involves checking subsets of the system. Acceptance, Function and System testing is performed upon the entire system.
Reference: Glenford J.,
"The Art of Software Testing", Meyers, John Wiley & Sons, 1979
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Integration Testing
Bottom-Up Testing
A13. Testing 5
System Testing
System - Requirements
A13. Testing 6
Unit Test (Black & White box techniques) discovers errors in individual modules requires coding (& testing) of driver routines
Top-Down Testing
Main module & immediate subordinate routines are tested first requires coding of routine stubs to simulate lower level routines system developed as a skeleton
Does not test the system functions Compares the system with its objectives, (system behavior) External Specification not used to compose the test cases (eliminates or reduces possible conflict of goals) System test cases are derived from the user documentation and requirements Compares user doc to program objectives No general system test-case-design procedure exists
Program Requirements User Documentation
Sandwich Integration
combination of top-down & bottom-up testing
Big Bang
No integration testing modules developed alone All modules are connected together at once
Program
External Specifications
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Function Testing
System - Specifications
Functional Verification Testing Establishes Level of Confidence
A13. Testing 7
Acceptance Testing
System - Users
A13. Testing 8
Tests the program against the current needs of the users and its original objectives. Usually performed by the end user (customer) Contract may require, as part of acceptance test:
performance tests (throughput, statistics collection, ...) stress tests (system limits)
Proof of Correctness
If performed by system developers may consist of (alpha), (beta) testing
Program Requirements User Documentation
Checks that the system satisfies its external specification Entire system is viewed as a "Black Box" Techniques:
Equivalence Partitioning Boundary-value Analysis Cause-Effect Graphing
Program
Program Requirements
Program
User Documentation
External Specifications
External Specifications
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Testing Experiment
Program
A13. Testing 9
Exhaustive Testing
Example
32 Bit Integer Component
A13. Testing 10
Program reads 3 integer values from a line. The 3 values represent the lengths of the sides of a triangle. The program outputs whether the triangle is equilateral, isosceles, or scalene. Write a set of test cases which would adequately test this program!
Output
Practical Limitations
Test Cases
How long will it take to try all possible inputs at a rate of one test/second?
Valid scalene triangle. Valid equilateral triangle. Valid Isosceles triangle. All possible permutations of Isosceles triangles (e.g. (3,3,4) (3,4,3) (4,3,3)) One side having a zero value. One side having a negative value. 1 Degenerate Triangle (e.g. 1-Dim (1,2,3) All possible permutations of Degenerate Triangles (e.g. (1,2,3) (3,1,2) (1,3,2)) Invalid Triangle (e.g. (1,2,4)) All possible permutations of invalid triangles. All sides = 0. Non-integer values. Incorrect number of sides ...
232 tests * 1 second / test = 232 seconds
= 232 / (60 * 60 * 24 * 365) years
2 3
> 232 / (26 * 26 * 25 * 29 ) years = 232 / 226 years
= 26 years = 64 years
Exhaustive Testing cannot be performed!
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Testing Principles
General Heuristics
A13. Testing 11
Testing Mechanics
Testing components
A13. Testing 12
The expected output for each test case should be defined in advance of the actual testing. The test output should be thoroughly inspected. Test cases must be written for invalid & unexpected, as well as valid and expected input conditions. Test cases should be saved and documented for use during the maintenance / modification phase of the life cycle. New test cases must be added as new errors are discovered. The test cases must be a demanding exercise of the component under test. Tests should be carried out by a third party independent tester, developer engineers should not privatize testing due to conflict of interest Testing must be planned as the system is being developed, NOT after coding.
Driver Test Case Inputs
a
b
Valid Test Outputs
Routine X Required by X but NOT coded
c
d
Stub
Component Under Testing
Drivers
Test harness
Goal of Testing
Perform testing to ensure that the Perform testing to ensure that the probability of program/system failure due probability of program/system failure due to undiscovered errors is acceptably small. to undiscovered errors is acceptably small.
Stubs
Scaffold Code
No method (Black/White Box, etc.) can be used to detect all errors. Errors may exist due to a testing error instead of a program error. A finite number of test cases must be chosen to maximize the probability of locating errors.
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
White Box Testing
Structural Testing
A13. Testing Box: 13
White Logic Testing
Logic Coverage
A13. Testing 14
Exercise of Source code and internal data structures Test cases are derived from analysis of internal module logic and external module specifications Logic Coverage (condition/decision testing)
Statement Coverage Decision Coverage Condition Coverage Decision/Condition Coverage Multiple Condition Coverage
Statement Coverage
Every statement is executed at least once.
Decision Coverage
Each decision is tested for TRUE & FALSE. correctness of conditions within the decisions are NOT tested
Condition Coverage
Each condition in a decision takes on all possible outcomes at least once. Does not necessarily test all decision outcomes. Test cases do not take into account how the conditions affect the decisions.
Path Coverage
Control Flow Testing
Correct I/O relationships are verified using both :
Decision/Condition Coverage
Satisfies both decision coverage and condition coverage. Does NOT necessarily test all possible combinations of conditions in a decision.
Functional Description and actual implementation
Multiple Condition Coverage
Test all possible combinations of conditions in a decision Does not test all possible combinations of decision branches.
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
White Box: Path Testing
Control Flow Graph
A13. Testing 15
Test Path Determination
Independent Path
A13. Testing 16
Node: sequence of statements ending in a branch Arc: transfer of control Exercise a program by testing all possible execution paths through the code. Method
1. Enumerate the paths to be tested 2. Find the Input Domain of each 3. Select 1 or more test cases from domains
Path Testing
any path that introduces at least one new set of processing statements (nodes), i.e. it must traverse an edge not previously covered. Independent Paths:
1. 1 - 2 - 6 2. 1 - 2 - 3 - 5 - 2 - 6 3. 1 - 2 - 3 - 4 - 5 - 2 - 6
A
B
1 2 3 4 5
Problem: Loops ( number of paths)
Paths: ABC; ABBC; AB ... BC
Cyclomatic Complexity C
Solution:
Restrict loop to N iterations Select small number of paths that yield reasonable testing.
upper bound on the number of independent paths, i.e. number of tests that must be executed in order to cover all statements. CC = edges - Nodes + 2
=E-N+2 = 7-6+2 = 3 = Predicate Nodes + 1 =P+1 =2+1 =3
Exhaustive Path Testing (impossible)
6
(analogue of exhaustive input testing) requires executing the total number of ways of going from the top of the graph to the bottom approx. 100 trillion, 1020 - 520 + 519 +. . . + 51 where 5 = number of unique paths assuming all decisions are independent of each other specification errors could still exist does not detect missing paths does not check data-dependent errors
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Computer Science Dept Va Tech Aug., 2001
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Path Input Domains
Input Domain Subset
Domain for Path: ABDEAF A B
A13. Testing 17
Reverse Execution
Reverse execution of a decision
A13. Testing 18
Input Domain Y <= 100 Y > 100
Reverse execution of an assignment
Y > 50
F C E F D
Y > 100
T
Y = Y + 50 ; Y > 100
Reverse execution of a sequence of decisions
Collected decisions are connected logically by AND.
Reverse Path Analysis
Input Domain
(Y > 50) && (Y <= 100) Y > 50
(Y > 100) && (Y > 50) T
-> Y > 100
Recreate the test Recreate the test data by 'tracing' the data by 'tracing' the path in reverse, path in reverse, collecting the collecting the conditions on the conditions on the input variables. input variables.
? ?
Intro Data Structures & SE
?
Y <= 100 F Y > 100
Y > 100 T
Computer Science Dept Va Tech Aug., 2001
1995-2001 Barnette ND, McQuain WD
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Reverse Path Test Example
Test Component
A13. Testing 19
Reverse Path Test Example (cont)A13. Testing 20
Test Path: 1 2 3 4 5 2 6 Reverse Path Execution
Computes Z = XY where X, Y are nonnegative integers
cin >> X >> Y ; Z=1;
(6) (2)
Y=0
1
cin >> X >> Y ; Z=1;
(5)
Y=Y/2 Y/2=0 While Y != 0 T
While Y != 0
T Y % 2 == 1 T
(4) (3)
Y / 2 = 0 && Y%2=1
2
3
Y % 2 == 1 T
(2)
Y / 2 = 0 && Y % 2 = 1 && Y <> 0 Y=Y/2; X=X*X;
Z=Z*X;
Z=Z*X; Y=Y%2; X=X*X;
4
cout << Z ;
(1)
5
6
cout << Z ;
Test Case: Y = 1 The input domain is bounded by the accumulated conditions.
Algorithm:
( x 2 ) (y/2) if y is even : x = if y is odd : x ( x 2 ) ( y -1) / 2
y
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD Computer Science Dept Va Tech Aug., 2001
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Testing Reliability
Question:
A13. Testing 21
Mutation Analysis
A13. Testing 22
The purpose of Mutation Analysis is to test the test suite. Original Mutant
When to stop testing? When no more errors exist. Impossible to ascertain. (1) How reliable is the set of test cases?
Data Domain
Answer:
(2) How reliable is the software being developed?
Time Domain
RELIABILITY Data Domain Time Domain
Shooman Jelinski-Moranda Musa
Coverage Mutation Analysis Error Seeding
Time Domain Reliability
MTBF : mean time between failures MTTF : mean time to failure MTTR: mean time to repair MTBF = MTTF + MTTR Availability = MTTF / (MTTF + MTTR) * 100 Estimate Methods: 1. Predictions based on calendar time 2. Predictions based on CPU time
Mutate Code to determine the adequacy of the test data. Determines whether all deliberately introduced (mutant) errors are detected by the original test cases.
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Mutation Analysis Process
Mutation Testing Diagram
A13. Testing 23
PROGRAM
Error Seeding
Error Scattergram Graph
A13. Testing 24
Test Cases Testing Expand Test Cases
No
Mutations
Mutation Testing Process
all Mutations discovered?
Yes
1. Program P is executed for test case T 2. If errors occur test case T has succeeded
Testing complete
Errors are corrected & retested until no errors with test case T are observed.
3. Program is Mutated P' 4. Mutant P' is executed for test case T
IF no errors are found { test case T is inadequate; further testing is required; // ERROR SEEDING new test cases are added & step 3 is repeated until all mutations are discovered; entire process is started again at step 1 with the new test cases ELSE // all mutations located by tests T T is adequate and no further testing is required. (mutations)
Technique
Estimate of the number of original undiscovered errors remaining in a system.
1. Intentionally introduce (seed) errors into the source code. 2. Execute test cases upon source code. 3. Count the number of seeded errors & original errors (unseeded errors) discovered. 4. Estimate the total number of original errors
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD
Error Seeding Process
Testing Subset
A13. Testing 25
Assume there are N undiscovered errors present in the system. Add S seeded errors to the system. Test cases discover: TS seeded errors TN Hypothesis: nonseeded (original) errors
Test Efficiency: T /S = E
s
= fraction of discovered errors
Computer Science Dept Va Tech Aug., 2001
Intro Data Structures & SE
1995-2001 Barnette ND, McQuain WD