Ch03-automation

Ch03-automation - Introduction to Software Testing (2nd...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Introduction to Software Testing (2nd edition) Chapter 3 Test Automation Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/softwaretest/ First version, 9 September 2011 What is Test Automation? The use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions Reduces cost Reduces human error Reduces variance in test quality from different individuals Significantly reduces the cost of regression testing Introduction to Software Testing, Edition 2 (Ch 3) © Ammann & Offutt 2 1 Software Testability The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met Plainly speaking – how hard it is to find faults in the software Testability is determined by two practical problems – How to provide the test values to the software – How to observe the results of test execution Introduction to Software Testing, Edition 2 (Ch 3) © Ammann & Offutt 3 Observability and Controllability Observability How easy it is to observe the behavior of a program in terms of its outputs, effects on the environment and other hardware and software components – Software that affects hardware devices, databases, or remote files have low observability Controllability How easy it is to provide a program with the needed inputs, in terms of values, operations, and behaviors – Easy to control software with inputs from keyboards – Inputs from hardware sensors or distributed software is harder Data abstraction reduces controllability and observability Introduction to Software Testing, Edition 2 (Ch 3) © Ammann & Offutt 4 2 Components of a Test Case A test case is a multipart artifact with a definite structure Test case values The values that directly satisfy one test requirement Expected results The result that will be produced when executing the test if the program satisfies it intended behavior Introduction to Software Testing, Edition 2 (Ch 3) © Ammann & Offutt 5 Affecting Controllability and Observability Prefix values Any inputs necessary to put the software into the appropriate state to receive the test case values Postfix values Any inputs that need to be sent to the software after the test case values 1. Verification Values : Values necessary to see the results of the test case values 2. Exit Commands : Values needed to terminate the program or otherwise return it to a stable state Executable test script A test case that is prepared in a form to be executed automatically on the test software and produce a report Introduction to Software Testing, Edition 2 (Ch 3) © Ammann & Offutt 6 3 A Test Automation Framework A set of assumptions, concepts and tools that support test automation Introduction to Software Testing, Edition 2 (Ch 3) © Ammann & Offutt 7 What is JUnit? Open source Java testing framework used to write and run repeatable automated tests JUnit is open source (junit.org) A structure for writing test drivers JUnit features include: – – – – Assertions for testing expected results Test features for sharing common test data Test suites for easily organizing and running tests Graphical and textual test runners JUnit is widely used in industry JUnit can be used as stand alone Java programs (from the command line) or within an IDE such as Eclipse Introduction to Software Testing (Ch 1) © Ammann & Offutt 8 4 JUnit Tests JUnit can be used to test … – … an entire object – … part of an object – a method or some interacting methods – … interaction between several objects It is primarily for unit and integration testing, not system testing Each test is embedded into one test method A test class contains one or more test methods Test classes include : – A test runner to run the tests (main()) – A collection of test methods – Methods to set up the state before and update the state after each test and before and after all tests Get started at junit.org Introduction to Software Testing (Ch 1) © Ammann & Offutt 9 Writing Tests for JUnit Need to use the methods of the junit.framework.assert class – javadoc gives a complete description of its capabilities Each test method checks a condition (assertion) and reports to the test runner whether the test failed or succeeded The test runner uses the result to report to the user (in command line mode) or update the display (in an IDE) All of the methods return void A few representative methods of junit.framework.assert – – – – – assertTrue (boolean) assertTrue (String, boolean) assertEquals (Object, Object) assertNull (Object) Fail (String) Introduction to Software Testing (Ch 1) © Ammann & Offutt 10 5 Sample Assertions static void assertEquals (boolean expected, boolean actual) Asserts that two booleans are equal static void assertEquals (byte expected, byte actual) Asserts that two bytes are equal static void assertEquals (char expected, char actual) Asserts that two chars are equal static void assertEquals (double expected, double actual, double delta) Asserts that two doubles are equal, within a delta static void assertEquals (float expected, float actual, float delta) Asserts that two floats are equal, within a delta static void assertEquals (int expected, int actual) Asserts that two ints are equal For a complete list, see – http://junit.sourceforge.net/javadoc/org/junit/Assert.html Introduction to Software Testing (Ch 1) © Ammann & Offutt 11 How to Write A Test Case You may occasionally see old versions of JUnit tests – – Major change in syntax and features in JUnit 4.0 Backwards compatible (JUnit 3.X tests still work) In JUnit 3.X 1. 2. 3. 4. import junit.framework.* extend TestCase name the test methods with a prefix of ‘test’ validate conditions using one of the several assert methods In JUnit 4.0 and later: – – – – – Do not extend from Junit.framework.TestCase Do not prefix the test method with “test” Use one of the assert methods Run the test using JUnit4TestAdapter @NAME syntax introduced We will focus entirely on JUnit 4.X Introduction to Software Testing (Ch 1) © Ammann & Offutt 12 6 JUnit Test Fixtures A test fixture is the state of the test – Objects and variables that are used by more than one test – Initializations (prefix values) – Reset values (postfix values) Different tests can use the objects without sharing the state Objects used in test fixtures should be declared as instance variables They should be initialized in a @Before method Can be deallocated or reset in an @After method Introduction to Software Testing (Ch 1) © Ammann & Offutt 13 Example JUnit Test Case public class Calc { static public long add (int a, int b) { import org.junit.Test return a + b; import static org.junit.Assert.*; } public class calcTest } { private Calc calc; @Test public void testAdd() { // Calc().add() returns long, // so we must cast 5 assertEquals ((long) 5, new Calc().add (2, 3)); } } Introduction to Software Testing (Ch 1) © Ammann & Offutt 14 7 Introduction to Software Testing (Ch 1) © Ammann & Offutt 8 16 } ;llun = kcats { { { { )(tseThcaEretfAnur diov cilbup retfA@ tset hcae retfa nur era sdohtem retfA@ // xatnys retfA@ gnisu dohtem nwod-raet // xatnys retfA@ gnisu dohtem nwod-raet // xatnys retfA@ gnisu dohtem nwod-raet // xatnys retfA@ gnisu dohtem nwod-raet // method (postfix) : • Post-test teardown } ;)(kcatS wen = kcats ;)(kcatS wen = kcats ;)(kcatS wen = kcats { )(tseThcaEerofeBnur diov cilbup erofeB@ tset hcae erofeb nur era sdohtem erofeB@ // xatnys erofeB@ gnisu dohtem )xiferp( putes // // ;kcats kcatS etavirp (prefix) : • Pre-test setup method ;retpadAtseT4tinUJ.krowemarf.tinuj tropmi ;retpadAtseT4tinUJ.krowemarf.tinuj tropmi ;retpadAtseT4tinUJ.krowemarf.tinuj tropmi ;retpadAtseT4tinUJ.krowemarf.tinuj tropmi ;slauqEtressa.tressA.tinuj.gro citats tropmi ;tseT.tinuj.gro tropmi ;erofeB.tinuj.gro tropmi ;erofeB.tinuj.gro tropmi ;erofeB.tinuj.gro tropmi ;retfA.tinuj.gro tropmi Classes to import : Stack Test Class } } } } } } } } ;eurt nruter ;eurt nruter ;eurt nruter ;eurt nruter } } } } ;esllaf nruter )llllun == ]ii[stnemelle( fii ;es af nruter ) un == ] [stneme e( f ;esllaf nruter )llllun == ]ii[stnemelle( fii ;es af nruter ) un == ] [stneme e( f } { { { { ;)(gnirtSot.fub nruter )++ii ;eziis < ii ;0 = ii tnii( rof ) + + ; e z s < ; 0 = t n ( r of )++ii ;eziis < ii ;0 = ii tnii( rof ) + + ; e z s < ; 0 = t n ( r of ;)"}"( dneppa.fub ;esllaf nruter )htgnell.stnemelle =! eziis( fii ;es af nruter )htgne .stneme e =! ez s( f ;esllaf nruter )htgnell.stnemelle =! eziis( fii ;es af nruter )htgne .stneme e =! ez s( f } ;esllaf nruter )llllun == stneme e ;es af nruter ) un == stneme ;esllaf nruter )llllun == stnemelle( f ;es af nruter ) un == stneme ;lle((gfnirtSot.] i [stnemele( dneppa.fub )e( fii ) ( fii { { { { ;)" ,"( dneppa.fub )(kOper naelloob ciillbup )(kOper nae oob c bup )(kOper naelloob ciillbup )(kOper nae oob c bup ))1-ezis( < i( fi { )--i ;0 => i ;1-ezis = i tni( rof ;)"{"( reffuBgnirtS wen = fub reffuBgnirtS .mottob eht ot pot eht morf kcatS siht fo // noitatneserper gnirtS eht snruteR :STCEFFE // { )(gnirtSot gnirtS cilbup { kcatS ssalc cilbup Introduction to Software Testing (Ch 1) © Ammann & Offutt 15 Testing the Immutable Stack Class Introduction to Software Testing (Ch 1) © Ammann & Offutt 9 18 } } } } ;)tluser ,eurt( slauqEtressa ;)(kOper.kcats = tluser naeloob ;)(pot.kcats ;)(pot.kcats ;)(pot.kcats ;)(pot.kcats ;))1( regetnI wen( hsup.kcats = kcats @Test public void testRepOkD() { } ;)tluser ,eurt( slauqEtressa ;))1( regetnI wen( hsup.kcats = kcats kcats ;)(pop.kcats = ;)tluser ,eurt( slauqEtressa ;)tluser ,eurt( slauqEtressa ;)tluser ,eurt( slauqEtressa ;)tluser ,eurt( slauqEtressa ;)(kOper.kcats = tluser naeloob ;)(kOper.kcats = tluser naeloob kcats ;))1( regetnI wen( hsup.kcats = } @Test public void testRepOkB() { } @Test public void testRepOkC() { ;)tluser ,eurt( slauqEtressa ;)tluser ,eurt( slauqEtressa ;)tluser ,eurt( slauqEtressa ;)tluser ,eurt( slauqEtressa ;)(kOper.kcats = tluser naeloob { )(AkOpeRtset diov cilbup tseT@ Stack Test Cases (2) Introduction to Software Testing (Ch 1) © Ammann & Offutt 17 } } ;)tluser ,eurt( slauqEtressa With automation, small tests allow us to more easily identify failures … ;)(kOper.kcats = tluser ;)(pot.kcats ;)(pot.kcats ;)(pot.kcats ;)(pot.kcats ;))1( regetnI wen( hsup.kcats = kcats ;)tluser ,eurt( slauqEtressa Without automation, large tests have the advantage of reducing costs of running many tests ;)(kOper.kcats = tluser ;)(kOper.kcats = tluser ;)(kOper.kcats = tluser ;)(pop.kcats = kcats ;)tluser ,eurt( slauqEtressa ;)(kOper.kcats = tluser ; regetnI wen( hsup.kcats = ;))1( regetnI wen( hsup.kcats = k kcats ;)tluser ,eurt( slauqEtressa ;)(kOper.kcats = tluser naeloob { { { { A problem with this test is that it actually combines four separate tests in one method )(kOpeRtset diov cilbup tseT@ } ;))(gnirtSot.kcats ,"}1 ,2{"( slauqEtressa ;))(gnirtSot.kcats ,"}1 ,2{"( slauqEtressa ;))(gnirtSot.kcats ,"}1 ,2{"( slauqEtressa ;))2( regetnI wen( hsup.kcats = kcats ;))1( regetnI wen( hsup.kcats = kcats { )(gnirtSoTtset diov cilbup tseT@ )(gnirtSoTtset diov cilbup tseT@ )(gnirtSoTtset diov cilbup tseT@ )(gnirtSoTtset diov cilbup tseT@ Stack Test Cases Introduction to Software Testing (Ch 1) © Ammann & Offutt 10 20 } } } } } ;)ssalc.stseTllA( retpadAtseT4tinUJ wen nruter { )(etius tseT.krowemarf.tinuj citats cilbup .tnA ro srennuR tseT 3 tinUJ gnisu nehw sfpleh dohtem )(etius ehT // } ;))(etius( nur.rennuRtseT.iutxet.tinuj { )sgra ][gnirtS( niam diov citats cilbup .liaf yna fi retset eht sllet taht rennur tset // .liaf yna fi retset eht sllet taht rennur tset // .liaf yna fi retset eht sllet taht rennur tset // .liaf yna fi retset eht sllet taht rennur tset // a setucexe ssalc tset sihT .)(niam ni snigeb noitucexE // { stseTllA ssalc cilbup .ereh sessalc tset ddA // .ereh sessalc tset ddA // .ereh sessalc tset ddA // .ereh sessalc tset ddA // )} ssalc.tseTkcatS {( sessalCetiuS.etiuS@ )} ssalc.tseTkcatS {( sessalCetiuS.etiuS@ )} ssalc.tseTkcatS {( sessalCetiuS.etiuS@ )} ssalc.tseTkcatS {( sessalCetiuS.etiuS@ )ssalc.etiuS( htiWnuR@ .margorp eht ni sessalc tset eht fo lla seralced noitces sihT // ;retpadAtseT4tinUJ.krowemarf.tinuj tropmi ;etiuS.srennur.tinuj.gro tropmi ;etiuS.srennur.tinuj.gro tropmi ;etiuS.srennur.tinuj.gro tropmi ;etiuS.srennur.tinuj.gro tropmi ;htiWnuR.rennur.tinuj.gro tropmi AllTests Introduction to Software Testing (Ch 1) © Ammann & Offutt 19 We need a main() for command line execution … This is all we need to run JUnit in an IDE (like Eclipse) Running from a Command Line How to Run Tests JUnit provides test drivers – Character-based test driver runs from the command line – GUI-based test driver-junit.swingui.TestRunner • Allows programmer to specify the test class to run • Creates a “Run” button If a test fails, JUnit gives the location of the failure and any exceptions that were thrown Introduction to Software Testing (Ch 1) © Ammann & Offutt 21 JUnit Resources Some JUnit tutorials – http://open.ncsu.edu/se/tutorials/junit/ (Laurie Williams, Dright Ho, and Sarah Smith ) – http://www.laliluna.de/eclipse-junit-testing-tutorial.html (Sascha Wolski and Sebastian Hennebrueder) – http://www.diasparsoftware.com/template.php?content=jUnitStarterGuide (Diaspar software) – http://www.clarkware.com/articles/JUnitPrimer.html (Clarkware consulting) JUnit: Download, Documentation – http://www.junit.org/ Introduction to Software Testing (Ch 1) © Ammann & Offutt 22 11 Summary The only way to make testing efficient as well as effective is to automate as much as possible JUnit provides a very simple way to automate our unit tests It is no “silver bullet” however … it does not solve the hard problem of testing : What test values to use ? • This is test design … the purpose of test criteria Introduction to Software Testing (Ch 1) © Ammann & Offutt 23 12 ...
View Full Document

Ask a homework question - tutors are online