Black box software testing: A course by Cem Kaner & James Bach
Introduction 2: The measurement problem and the impossibility of complete testing
- Video lecture Part 1 [29:37] Part 2 [26:44]
- Lecture slides (editable PPT's)
- Multiple choice review questions [doc] [grading notes]
- Essay test questions
- An orientation exercise. (A pretest to prime the student for some issues that will come up in lecture.)
- A classroom activity (Calculating different coverage numbers for small program fragments)
- Examples of calculations
- Assigned reading: Hoffman, Exhausting your test options.
- Assigned reading: Kaner, The impossibility of complete testing.
- Assigned reading: Marick, How to misuse code coverage.
- Assigned reading: Simmons, When will we be done testing? Software defect arrival modeling using the Weibull distribution.
It's impossible to fully test a program. There are too many inputs, too many combinations, too many paths, too many places where too many types of interrupts can impact the execution of hte program, too many ways the program can be used, and too many interesting ways the program can fail. In the face of an infinitely large testing task, we have to treat with skepticism the statements that some people make that the testing project must always do this or always deliver that. In the face of an infinitely large task, everything is a tradeoff--work spent on one task is work not allocated to another. The proper allocation of resources to tasks and deliverables has to be a function of the information objectives of the project at hand.
Several metrics appear to check how close we are to completeness of testing, and thereby define testing completeness. Coverage measures are an example; so are defect arrival rate probability models. If "complete testing" means that there are no remaining unknown bugs, then these approaches do not measure completeness of testing. In addition, there are predictable risks of using these metrics. Coverage measurement has value, but not as an indicator of how close we are to completion of testing.
We are setting up a mailing list for announcements about this course and, perhaps, a tightly focused and moderated discussion of how to teach it or self-study with it. (This won't be a general, high-traffic, intro-to-testing discussion.) If you're interested in the course, please sign up by sending us an email. We will NOT share your email address with third parties or send commercial advertising to you.
We are publishing this course under a Creative Commons license that allows you to freely reuse and distribute the materials and to modify the slides and associated printable materials (but not the videos). We would be appreciate a few mirror sites, to reduce the growing burden on our servers. If you can help in this way, or any other way, please send a note to Cem Kaner.
