Black Box Software Testing: Spring 2005

Cem Kaner & James Bach

BBST has been evolving since 1993, with substantial contributions by Hung Nguyen, Doug Hoffman, Brian Lawrence, Bob Johnson, Sam Guckenheimer, Pat McGee, Ajay Jha, and students in about 100 teachings of the commercial version of this course.

The tables below are our project tracking charts. Our public user interface to the course is at this page, not here.

In our tracking chart, we break the course into learning units, track several tasks per unit. Inside a cell:

  • A number refers to the draft (first, second, etc.). Several units' slides are past 20th draft. We arbitrarily number longstanding slide sets in good shape as 5th draft, with lower draft numbers for sets that have survived a lot of drafts but aren't yet well polished.
  • An item can be designated with P (publicly presented at a conference or a customer site that provided extended feedback), R (peer reviewed), T (taught from in an academic course), E (reviewed by an education specialist), with extent to which it needs change of L (low), M (medium), H (high) and C (we think it is pretty much complete). Thus a mature item might be coded as 5PRTEC.

Click on a status code (such as 5PR) to load the document in its native file format (PowerPoint, Word, Open Office, etc.) Click on [PDF] for the Adobe PDF format. NOTE: Because of incompatibilities between MS PowerPoint on the Mac and Windows, graphics on a few slides will not display on Windows machines. We'll fix these eventually. For now, the correct graphics are available to both platforms in the [PDF] files.

FULL SET OF LECTURE NOTES [PDF] [NOTE: These are the 2004 notes. Go to the relevant section for the 2005 updates. The 2005 files are changing every day.]

Review Questions:

COURSE SECTION
lecture slides
Examples
Review Questions
Video
Orientation
Class Activity
Assignments
Overview Readings
syllabus
3TL PDF                
oracles
5PRTL PDF Examples 2H

EssayQues 1M

MultipleChoice 1M (Notes)

full  video lecture 2TC Orientation 1T PDF Activity 1M   Overview Readings
complete testing
5PRTM PDF Examples 2H

EssayQues 2TM

Multiple Choice 1M  (Notes)

video lecture 1TC: First Half Second half

Orientation 1T PDF Activity 1M   Overview Readings
bug advocacy
5PRTC PDF  

bugs & errors EssayQues 2TM

advocacy EssayQues 1TH

Multiple Choice 1M (Notes)

video lecture 1TC Orientation 1T

Activity 1M

Activity 1M

5PRTL PDF Overview Readings
bug advocacy 2: quality-related costs
5PRTL PDF  

EssayQues 1TH

Multiple Choice 1M (Notes)

video lecture 1TC       Overview Readings
bug advocacy 3: effective writing

deal with objections 5PRTC PDF

editing 5PRTC PDF

 

deal with objections EssayQues 1H

editing EssayQues 1H

     

editing Assignment 5PTC PDF

rubric PDF

Overview Readings
bug advocacy 4: bug tracking system mission; your credibility
5PRM PDF   EssayQues 0H         Overview Readings

domain

5PRTL PDF Examples 1H  EssayQues 2TM

Part A  Part B Part C Part D Part E Solutions

 

Activity 1M

Grading notes

domain assignment 5TC PDF

Overview Readings
5PRTL PDF EssayQues 2TM Part A  Part B Part C Hockey example Triangle example Foodvan Example    

domain assignment 5TC PDF

grading notes

risk
bug taxonomies 5PRTM PDF Examples 1H

EssayQues 2TM

Part 1 Part 2 Part 3 Part4 Part5       Overview Readings
quick tests Attacks 1H Interference 1H

EssayQues 2TM

Multiple Choice 1M (Notes)

  Activity 1M  (Notes) attacks assignment 3TL PDF
operational profiles   EssayQues 0H      
scenario
5PRTC PDF

Examples 1H

Multiple Choice 1M (Notes)

EssayQues 2TM full video lecture 1TC    First half   Second half   Activity 1M  (Notes)   Overview Readings
function
5PTM PDF Examples 1H EssayQues 1H lecture       Overview  
specification
5PRTM PDF Examples 1H

EssayQues 1TH

Questioning EssayQues 0H

Video: Part1 Part2 Part3 Part4   Activity 1M  (Notes)   Overview  
regression
5PRT PDF Examples 1H EssayQues 1TM Video: Part1 Part2 Part3       Overview  
user
3PTH PDF   EssayQues 1H            
stress
3PTH PDF Examples 1H EssayQues 1H            
test design
5PRTL PDF   EssayQues 1TM

video: First half, Second half  Exercise

  Activity 1M  (Notes)   Overview Readings
matrices
5PRTC PDF Examples 1H EssayQues 1H            
stochastic
5PTM PDF   EssayQues 1TH            
GUI automation
5PRTL PDF   EssayQues 2TM Video: Part1 Part2 Part3   Activity 1M GUI requirements assignment Overview  
high-volume automation
5PRTM PDF   EssayQues 1TM            
architecture
5PTRM PDF   EssayQues 1H            
exploratory
5PTRH PDF Examples 1H EssayQues 2TM            
paired exploration
3PTRL PDF   EssayQues 1H     Activity 1M      
multi-variable
5TM PDF Examples 1H EssayQues 2TH Video: Part1 Part2 Part3 Part4 Part5 Part6   Activity 1M  (Notes)   Overview www.pairwise.org
questioning strategies
Now incorporated in specification-based testing
test docs & requirements
5PRTM PDF  

DocTypes 1TH

Requirements 1TH

Video: Part1 Part2     docs assignment 1TM  PDF Overview  
scripting

5PRTL PDF  demon-stration [PDF]

  EssayQues 1TH Video: Part1 Part2 Demo       Overview  
test planning
5PRM PDF   EssayQues 1H            
career planning
3PH PDF   EssayQues 0H            
recruiting
5PRTC PDF   EssayQues 0H            
learning styles
3PM PDF   EssayQues 0H            
outsourcing
    EssayQues 0H            
legal issues
    EssayQues 0H            
mgmt issues
    EssayQues 0H            
measuring tester performance
    EssayQues 0H            
four schools of testing
    EssayQues 0H            
context analysis
    EssayQues 1T2            
research guide to testing
    EssayQues 0H            
intro to TDD
    EssayQues 0H            
lifecycle models
    EssayQues 2TM            
milestones in SDLC
    EssayQues 0H            
roles of test groups
    EssayQues 1H            
nature of quality
                 
state models
    EssayQues 1H            
testability
    EssayQues 0H            
testers' notebooks
    EssayQues 0H            
project resource estimation
    EssayQues 0H            
code coverage
  Examples 1H EssayQues 1TH            
measuring the extent of testing
5PRTL PDF   EssayQues 1TM            
measurement theory
5PRTL PDF   EssayQues 2TM            
status reports
4PRTM PDF   EssayQues 1TM            

Types of Materials

  • Lecture slides. These are the slides we display in class. Because so many versions of PowerPoint have lost or corrupted Slide Notes, we handle instructor notes separately.
  • Learning objectives. What we hope students will learn in in this Unit. Some, but not necessarily all, of the objectives are phrased in objective, measurable terms.
  • Suggested readings. A list of related papers or book chapters that students interested in this learning unit could consult.
  • Articles. These articles are provided at the site or are publicly available and linked from the site. A typical article might be a literature review of the technique (its history, theoretical underpinnings, and application) or a paper (or book chapter) that covers the same ground as a subsection of the slides but in more detail. This is like a textbook chapter that students read to supplement their lectures notes.
  • Key examples. These include descriptions of events in real projects that illustrate the technique well, and lab-created examples.
  • Instructor notes explain to another instructor why a given approach was taken, how to run certain activities or what to expect during them, what some alternate activities might be, and so on.
  • Context notes. A summary analysis of circumstances under which the technique appears to be effective or ineffective, and description of other techniques that work well in conjunction with this one.
  • Orientation. A pretest or other activity to orient learners to the issues addressed by this technique.
  • Assignments. Homework (simple to complex assignments).
  • Class Activity. Description of an in-class lab or seminar
  • Factual review.Memorized-concept (basic factual) questions to help learners identify whether they have at least a superficial understanding of the materials.
  • Skill drill. Drill-type questions that practice learners on calculations or other applications of the technique.
  • Exam questions. Sample exam questions.
  • Mastery. Authentic activities that can be assigned to the learner and evaluated by an expert as competent examples of the use of the technique (or not, and if not, what appears problematic).
  • Task / skill analysis. A description of the key skills involved in applying the technique.
  • Video lecture. A videotape of the full lecture or lectures for the technique.
  • Video clips. Short video segments that describe or demonstrate some part of a learning unit. Ideally, a video clip will film someone who has first hand experience with the example.
  • Topic links. Links to relevant websites and individual documents that can be reached online.