schuler-issta-2009 - Efcient Mutation Testing by Checking...

Info icon This preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon
Efficient Mutation Testing by Checking Invariant Violations David Schuler, Valentin Dallmeier, and Andreas Zeller Saarland University, Saarbrücken, Germany { ds, dallmeier, zeller } @cs.uni-saarland.de ABSTRACT Mutation testing measures the adequacy of a test suite by seeding artificial defects (mutations) into a program. If a mutation is not detected by the test suite, this usually means that the test suite is not adequate. However, it may also be that the mutant keeps the program’s semantics unchanged— and thus cannot be detected by any test. Such equivalent mutants have to be eliminated manually, which is tedious. We assess the impact of mutations by checking dynamic invariants. In an evaluation of our JAVALANCHE framework on seven industrial-size programs, we found that mutations that violate invariants are significantly more likely to be de- tectable by a test suite. As a consequence, mutations with impact on invariants should be focused upon when improv- ing test suites. With less than 3% of equivalent mutants, our approach provides an efficient, precise, and fully automatic measure of the adequacy of a test suite. Categories and Subject Descriptors D.2.5 [ Software ]: Software Engineering— Testing and De- bugging General Terms Experimentation Keywords Dynamic Invariants, Mutation Testing 1. INTRODUCTION How do we know a test suite is adequate in finding de- fects? Among the best ways is mutation testing: seeding defects into a program and checking whether the test suite finds them. Such defects can be created automatically, us- ing a set of mutation operators to change (“mutate”) random program parts. A mutation that is not detected (“killed”) by the test suite indicates that the test suite was unable to Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ISSTA’09, July 19–23, 2009, Chicago, Illinois, USA. Copyright 2009 ACM 978-1-60558-338-9/09/07 ...$5.00. detect the seeded defect—and therefore is likely to miss sim- ilar, true defects in the code. Test managers can use such results to improve their test suites such that they detect these mutants. Mutation testing has been shown to be an effective mea- surement for the quality of a test suite [ 2 ] and superior to commonplace assessments such as coverage metrics [ 27 , 8 ]. However, mutation testing is known to be very expensive. A well-known issue is its large usage of computing resources. A less known, but far more significant cost, though, comes from the problem of equivalent mutants. These are mutants that leave the program’s overall semantics unchanged—and therefore cannot be caught by any test suite: The result of mutation testing—”surviving” mutations not found by the
Image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 2
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern