Chapter 26 - Chapter 26 Chapter Testing Bjarne Stroustrup

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Chapter 26 Chapter Testing Bjarne Stroustrup www.stroustrup.com/Programming Abstract Abstract This lecture is an introduction to the design and This testing of program units (such as functions and classes) for correctness. We discuss the use of interfaces and the selection of tests to run against them. We emphasize the importance of designing systems to simplify testing and testing from the start. Proving programs correct and performance problems are also briefly considered. are Stroustrup/Programming 2 Overview Overview Correctness, proofs, and testing Dependencies System tests Testing GUIs Resource management Unit and system tests Finding assumptions that do not hold Design for testing Performance Stroustrup/Programming 3 Correctness Correctness Questions to ask about a program You have to be able to reason about your code to have any real You certainty certainty Is your program correct? Is What makes you think so? What How sure are you? Why? Would you fly in a plane that depended on that code? Programming is generally unsystematic Debugging is generally unsystematic What are you willing to bet that you found the last bug? Related interesting questions Could the program run forever if the hardware didn’t fail? Does it always deliver its results in a reasonable time? Stroustrup/Programming 4 Proofs Proofs So why not just prove mathematically that our program is So correct? correct? It’s often too hard and/or takes too long Sometimes proofs are wrong too (even proofs produced by computers or Sometimes by experts!). by Computer arithmetic isn’t the same as “real” math—remember the Computer rounding and overflow errors we saw (due to finite and limited precision)? precision)? So we do what we can: follow good design principles, test, test, and then So test some more! test Stroustrup/Programming 5 Testing Testing “A systematic way to search for errors” Real testers use a lot of tools Unit test frameworks Static code analysis tools Fault injection tools … When done well, testing is a highly skilled and most valuable When activity activity “Test early and often” Whenever you write a function or a class, think of how you might test it Whenever you make a significant change, re-test Before you ship (even after the most minor change), re-test Stroustrup/Programming 6 Testing Testing Some useful sets of values to check (especially boundary cases): the empty set small sets large sets sets with extreme distributions sets where “what is of interest” happens near the ends sets with duplicate elements sets with even and with odd number of elements some sets generated using random numbers Stroustrup/Programming 7 Primitive test harness for binary_search() Primitive int a1[ ] = { 1,2,3,5,8,13,21 }; if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),1) == false) cout << "1 failed"; if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),5) == false) if cout << "2 failed"; cout if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),8) == false) cout << "3 failed"; if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),21) == false) cout << "4 failed"; if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),-7) == true) if cout << "5 failed"; cout if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),4) == true) if cout << "6 failed"; cout if (binary_search(a1,a1+sizeof(a1)/sizeof(*a1),22) == true) cout << "7 failed"; Stroustrup/Programming 8 A Better Test Harness (stil primitive) (stil Put the variables into a data file, e.g., with a format of Put e.g. { 27 7 { 1 2 3 5 8 13 21} 0 } meaning {test_number value {sequence} result} i.e., test #27 calls our binary_search to look for the value 7 in the sequence { 1 2 3 5 8 13 21} and checks that the result is 0 (false, that 13 is, not found). is, Now it’s (relatively) easy to write lots of test cases, or even write another Now program to generate a data file with lots of (random) cases. program Stroustrup/Programming 9 Dependencies Dependencies Basically we want every function to: have well-defined inputs have well-defined results not have dependencies on objects that are not its explicit inputs including any modifications to input parameters in a determinate amount of time (no infinite loops, please) in Hard to achieve in real life not use more resources than are available and appropriate E.g., time, memory, internet bandwidth, files, and locks Stroustrup/Programming 10 Dependencies Dependencies How many dependencies can you spot in this nonsense function? int do_dependent(int a, int& b) // messy function int messy // undisciplined dependencies undisciplined { int val ; cin>>val; vec[val] += 10; cout << a; b++; return b; } Stroustrup/Programming 11 Resource Management Resource What resources (memory, files, etc.) acquired may not always be What properly released in this nonsense function? properly void do_resources1(int a, int b, const char* s) // messy function void messy // undisciplined resource use undisciplined { FILE* f = fopen(s,"r"); // open file (C style) FILE* open int* p = new int[a]; // allocate some memory int* allocate if (b<=0) throw Bad_arg(); // maybe throw an exception if maybe int* q = new int[b]; // allocate some more memory int* allocate delete[ ] p; // deallocate the memory pointed to by p delete[ deallocate } Stroustrup/Programming 12 Better Resource Management Better // less messy function // less void do_resources2(int a, int b, const string& s) void { istream is(s.c_str(),"r"); // open file istream open vector<int>v1(a); // create vector (owning memory) vector<int>v1(a); create if (b<=0) throw Bad_arg(); // maybe throw an exception if // maybe vector<int> v2(b); // create another vector (owning memory) vector<int> create } Can do_resources2() leak anything? Stroustrup/Programming 13 Loops Loops Most errors occur at the ends, i.e., at the first case or the Most i.e. at last case. Can you spot 3 problems in this code? 4? 5? last int do_loop(vector<int>& vec) // messy function int messy // undisciplined loop undisciplined { int i; int sum; while(i<=vec.size()) sum+=v[i]; return sum; } Stroustrup/Programming 14 Buffer Overflow Buffer Really a special type of loop error, e.g., “storing more bytes than Really e.g. “storing will fit” into an array—where do the “extra bytes” go? do (probably not a good place) (probably The premiere tool of virus writers and “crackers” (evil hackers) Some vulnerable functions (best avoided): gets, scanf // these are the worst: avoid! // these sprintf strcat strcpy … Stroustrup/Programming 15 Buffer overflow Buffer Don’t avoid unsafe functions just as a fetish Understand what can go wrong and don’t just write equivalent code Even unsafe functions (e.g. strcpy()) have uses Even strcpy() have if you really want to copy a zero terminated string, you can’t do better than strcpy() – if just be sure about your “strings” (How?) just char buf[MAX]; char* read_line() // harmless? Mostly harmless? Avoid like the plague? // harmless? { int i = 0; char ch; while (cin.get(ch) && ch!='\n') buf(i++)=ch; buf[i+1]=0; return buf; } Stroustrup/Programming 16 Buffer overflow Buffer Don’t avoid unsafe functions just as a fetish Understand what can go wrong and don’t just write equivalent code Write simple and safe code string buf; getline(cin,buf); getline(cin,buf); // buf expands to hold the newline terminated input buf Stroustrup/Programming 17 Branching Branching In if and switch statements In if switch Are all alternatives covered? Are the right actions associated with the right conditions? Be careful with nested if and switch statements Be and statements The compiler ignores your indentation The Each time you nest you must deal with all possible alternatives Each level multiplies the number of alternatives (not just add) For switch statements For switch remember the default case and to break after each other case remember break unless you really meant to “fall through” unless Stroustrup/Programming 18 18 System Tests System Do unit tests first, then combinations of units, and so on, till we Do get to the whole system get Ideally, in isolation from other parts of the system Ideally, in a repeatable fashion What about testing GUI based applications? Control inversion makes GUI testing difficult Control Human behavior is not exactly repeatable Human Simulate user input from a test script Timing, forgetfulness, boredom, etc. Humans still needed at some point (only a human can evaluate “look and feel”) That way a test harness script takes the place of the human for many tests An excellent application of “layering” with well-defined interfaces An between the layers between Allows for portability of applications across GUI systems A GUI is often used as a lock-in mechanism Stroustrup/Programming 19 Testing Classes Testing A type of unit test A base class must be tested in combination with its derived classes but most class objects have state but Classes often depend on interactions among member functions Virtual functions Construction/initialization is the combined responsibility of several classes Private data is really useful here (beware of protected data members) Take our Shape class as an example: Take class Shape has several functions Shape has A Shape has a mutable state (we can add points, change color, etc.); that is, the effect of one function can affect the behavior of another function the Shape has virtual functions; that is, the behavior of a Shape depends on Shape what (if any) class has been derived from it what Shape is not an algorithm (why not?) Shape is A change to a Shape can have an effect on the screen (so maybe we still change need a human tester?) need Stroustrup/Programming 20 Finding assumptions that do not hold Finding For example, illegal input arguments Check before each call or at the beginning of the function Should never happen, but it does Depending on which code we can modify E.g., sqrt first checks that its argument is a non-negative value sqrt That can be difficult/problematic: Consider binary_search(a,b,v); Consider For forward iterators (e.g., for a list), we can’t test if a<b – no < operation for list), a<b For random-access iterators ,we can’t check if a and b are part of the same For sequence sequence The only perfect solution involves a run-time checking library Scanning the entire sequence to verify it’s sorted is much more work than Scanning actually doing the binary search actually // is v in [a:b) // The purpose of binary_search() is to be faster than linear search Sometimes, check in “debug/test mode” only Leave (only) affordable tests in production code Stroustrup/Programming 21 Design for Testing Design Use well-defined interfaces Have a way of representing operations as text to catch bad arguments before system testing Minimize dependencies and keep dependencies explicit so that they can be stored, analyzed and replayed Embed tests of unchecked assumptions (assertions) in the Embed calling and/or called code calling so that you can write tests for the use of these interfaces Define invariants, pre- and post conditions To make it easier to reason about the code Have a clear resource management strategy This will also minimize debugging! Stroustrup/Programming 22 Performance Performance Is it efficient enough? Is Note: Not “Is it as efficient as possible?” Note: Not Computers are fast: You’ll have to do millions of operations Computers to even notice (without using tools) to Accessing permanent data (on disc) repeatedly can be Accessing noticed noticed Accessing the web repeatedly can be noticed Accessing Time “interesting” test cases e.g., using time or clock() using or Repeat ≥3 times; should be ± 10% to be believable Stroustrup/Programming 23 Performance Performance What’s wrong with this? for (int i=0; i<strlen(s); ++i) { // do something with s[i] // do } It was part of an internet message log analyzer Used for files with many thousands of long log lines Stroustrup/Programming 24 Using clock() clock() int n = 10000000; // repeat do_somenting() n times int repeat clock_t t1 = clock(); if (t1 = = clock_t(-1)) { // clock_t(-1) means “clock() didn't work” if clock_t(-1) cerr << "sorry, no clock\n"; exit(1); } for (int i = 0; i<n; i++) do_something(); // timing loop for timing clock_t t2 = clock(); if (t2 = = clock_t(-1)) { cerr << "sorry, clock overflow\n"; exit(2); } cout << "do_something() " << n << " times took " << double(t2-t1)/CLOCKS_PER_SEC << " seconds " // scale result << scale << " (measurement granularity: " << CLOCKS_PER_SEC << " of a second)\n"; Stroustrup/Programming 25 ...
View Full Document

Ask a homework question - tutors are online