Lecture 8 Terms

A Hybrid Method

Shotgun testing is nevertheless interesting, because it tries lots of different inputs. It can be used along with input partition testing by applying the shotgun method to choose random input values within each partition.

Three Kinds of Black Box Methods

Systematic black box methods can be divided into three classes corresponding to these three kinds of information: 1. Input Coverage Tests 2. Output Coverage Tests 3. Functionality Coverage Tests

Black Box Method 1

Systematic functionality coverage testing

Black Box Method 2

Systematic input coverage testing

Shotgun Robustness Testing

Testing with random garbage inputs

Input Coverage

The second kind of black box testing. By analyzing all possible inputs allowed by the functional specifications, create test sets based on the analysis. The objective is to show software correctly handles all allowed inputs

Difference between Shotgun Testing and Boundary Value Testing

Unlike shotgun testing, boundary value testing is a systematic test method, because it has an easy way to choose test cases, and an easy way to know when we are done. (when all boundary values have been tested)

Input Coverage Methods

-Exhaustive -Input Partitioning -Shotgun -Boundary

Three Kinds of Information in Functional Specifications

1. The Intended Inputs 2. The corresponding intended actions 3. The corresponding intended outputs

Functionality Coverage Tests

Based on an analysis of the intended actions, with or without their inputs and outputs

Input Coverage Tests

Based on an analysis of the intended inputs, independent of their actions or outputs

Output Coverage Tests

Based on an analysis of the intended outputs, independent of their inputs or actions

Advantage to Black Box Methods

Black box methods can be done independently of the software on a large project, black box tests can be developed in parallel with the development of the software, saving time

Is Black box shotgun testing systematic?

Black box shotgun testing is still a black box method - we don't need the code to invent appropriate inputs at random. It is not really systematic - although there is a system for choosing test cases, there is no completion criterion

Why Boundary Values were Created

Black box testers crate boundary value tests to check that the program is robust with inputs on the edge

Shotgun Testing

Consists of choosing random values for inputs (with or without worrying about legality) over a large number of test runs. From then outputs are verified for legal inputs, and that program simply did not crash for illegal inputs. Illegal and Legal sets of inputs are set as separate sets of shotgun tests

Test Cases for Each Requirement

Each partitioned requirement is modeled as independent, separate test cases are created for each partitioned requirement

Third Advantage of Input Partition Testing

Easy to say when we are done (when we have run at least one representative test for each partition)

Boundary Values

Even when programs behave well with input values well outside their expected range, it is typical that failures come at the boundaries of the legal or expected range of values.

Experimental Design

Experiments should be designed such that each possible cause that may affect the outcome can be observed independently. Usual way to do this is to design the experiment in steps that only vary one "variable" at a time keeping everything else constant

Guidelines for Choosing Test Inputs

For test inputs, this principle means that we should help isolate failure causes, by as much as possible

Second Advantage of Input Partition Testing

Generally easy to identify a set of partitions given the functional specification

Fourth Advantage of Input Partition Testing

Gives us confidence that the program is at least capable of handling each different kind of input

Exhaustive Testing

Ideally input coverage testing should try every possible input to the program

Black Box Methods

In a black box method, test cases are solely based on the requirements, specification or (sometimes) design documents

Constant Test Cases

Keeping Everything constant between test cases, varying only one input value at a time

First Advantage of Input Partition Testing

Makes us think of intuitively for testing - that is, test the response to each kind of input

Input Partitioning

Partition all the possible inputs into equivalence classes which characterize the sets of inputs with something in common

Requirements Partitioning

Physically partition the functional specification into separate requirements

Functional Specifications

Can be formal (mathematical) or more often informal (in a natural language such as English)

Consistent Simple Input Value

Choosing the simplest input values possible in order not to introduce arbtitary variations

Covering Partitions

Partitions give us our test cases - all we must do now is design test cases to cover each partition

Functionality coverage

Partitions the functional specification into separate requirements to test

Input Robustness Testing

Robustness is the property that program doesn't crash or halt unexpectedly, no matter what the input. Two kind of robustness testing, shotgun robustness testing and boundary value robustness testing