Specification-Based Testing

Overview

THIS MATERIAL HAS BEEN UPDATED AND INCORPORATED IN THE TEST DESIGN LECTURES AT http://www.testingeducation.org/BBST/testdesign/. WE WILL REMOVE THIS SECTION IN MID-2012.

I define Specification-Based Testing broadly to include testing against any published claim about how the program is supposed to work. Thus, we might test against a comprehensive development specification, a user manual, a third-party manual, or a collection of emails and meeting notes that together define the product.

Specification-based testing is a style of testing (a style is a collection of test-related activities and techniques) focused on

• discovering what claims are made in the specifications and
• testing the product against them.

Course sections on spec-based testing often focus on analyzing fine details of the specification, such as ambiguity analysis or teasing out the logical relationships among variables to support combination testing. These are testworthy issues, but I think the biggest challenge in specification-based testing is figuring out what the specification says. Specifications are often hundreds or thousands of pages long. Organizing and reconciling its many treatments of the same issues, and its remarkably light treatments of some other issues, requires a conscious, active interaction with the set of documents that together make up the specification.

This section of the course considers the following questions:

• What is the specification?
• Of the many documents labeled "specification," which of the conflicting assertions in them are the correct (currently intended) descriptions of the product? There are also other documents, implicit specifications, that were not written as specifications of this particular product but that describe aspects of how the product must work (or how people will expect it to work).
• What does the specification say?
• We review active reading techniques for identifying and organizing complex sets of material.
• This includes review of questioning techniques, including context-free questions.
• We also apply the Heuristic Test Strategy Model to organize analysis of a complex specification.
• What are some techniques for critiquing the specification?
  • This is where we cover ambiguity analysis
• How to critique the specification?
• We look briefly at the process of conducting specification reviews
• How to derive tests from the specification?
• The main issue we consider here is coverage of the specification, including use of the traceability matrix
• What legal issues can help guide us in testing the specification?
• Written statements of fact in documents created by the vendor of a product are often interpreted as warranties--legally binding promises that the product will perform as described. Despite the potential legal liability, many software publishers invest minimal time checking the accuracy of their documentation.

Slides

Videos