A Course in Black Box Software Testing

Examples of Test Oracles

See lecture notes on oracles and test automation.

Copyright (c) Cem Kaner, 2004

An oracle is a mechanism for determining whether the program has passed or failed a test.

A complete oracle would have three capabilities and would carry them out perfectly:

One of the key problems with oracles is that they can only address a small subset of the inputs and outputs actually associated with any test. The tester might intentionally set the values of some variables, but all of the program's other variables have values too. Configuration settings, amount of available memory, and program options can also affect the test results. As a result, our evaluation of the test results (that we look at) in terms of the test inputs (that we set intentionally) is based on incomplete data, and may be incorrect.

Any of the oracle capabilities may be automated. For example, we might generate predictions for a test from previous test results on this program, from the behavior of a previous release of this program or a competitor's program, from a standard function or from a custom model. We might generate these by hand, by a tool that feeds input to the reference program and captures output or by something that combines automated and manual testing. We might instead generate predictions from specifications, advertised claims, regulatory requirements or other sources of information that require a human to evaluate the information in order to generate the prediction.

The following examples illustrate the use of oracles:

Copyright (c) Cem Kaner 2004

This work is licensed under the Creative Commons Attribution-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.

These notes are partially based on research that was supported by NSF Grant EIA-0113539 ITR/SY+PE: "Improving the Education of Software Testers." Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.