CSE 565

What is the process to find the basis paths in structured dynamic testing?

1. Select arbitrary path through graph 2. Flip first decision while keeping the test the same 3. Reset the first decision and flip the second 4. Continue until all decisions have been flipped

What is regression testing?

A non-functional/software quality testing methodology where it ensures previously developed and tested functions continue to work as specified after software modifications have been made

What is Configuration Testing?

A non-functional/software quality testing methodology where the tests focus on ensuring compatibility between required configurations (I.e. different OS, software versions, hardware, etc).

What is performance testing?

A non-functional:software quality testing methodology where the tester varies the load and stress on the system and sees how the system reacra

On waterfall development, when does testing start?

After all of the coding is complete.

What are the taxonomy categories called commonly used in defect based testing?

Beizer Generic Defect Taxonomy Categories

Does configuration testing test functional requirements, non-functional requirements, or both?

Both

Is black box behavioral/scenario based testing a validation, verification, or both activity?

Both

How is functional coverage demonstrated?

By mapping equivalence partitions to test cases

What is decision coverage?

Develop test cases such that all branches/test predicates get tested at least once. Decision coverage implies statement coverage.

What is multiple condition coverage?

Develop test cases such that all combinations of conditions in a decision are tested. This is the goal but difficult and at times impossible. This implies statement, decision, and decision/condition coverage.

What is decision/condition coverage?

Develop test cases such that each condition in a decision takes on all possible outcomes at least once and each decision takes on all possible outcomes at least once. Implies statement and decision coverage.

Why do defects cluster?

Due to complex code, programmer skill, etc.

What are some of the reasons why modifying existing software is such a high risk?

Error fixes, incorporation of new functions, code ripple effects, unintended feature interactions, changes in performance, resource sharing, etc

In agile development how often does testing happen?

Every day

What is a defect?

Fault in code made by an error

Is a 4-way DOE design full factorial or fractional factorial?

Fractional factorial

What are some of the drawbacks of pairwise/2-way tests?

May leave many functions untested and many software functions contain many parameters and factors

What is Orthagonal Defect Classification (ODC)?

Measurement method that uses the defect stream to provide measurability into the product and person.

What is the goal of DOE?

Minimize the number of tests we need to run.

What is an error?

Mistake made by human

Is black box static/requirement driven testing a validation, verification, or both activity?

Primarily verification

What is verification?

Showing we are building the program right/based on the requirements

What is a failure?

Software doesn't do what it's intended/supposed to. Caused by a defect.

What kind of testing is black box behavioral/scenario testing?

Specification/Functional Based Testing

What is Equivalence Partitioning (EP)?

Specification/Functional where you test different input classes

What is definition use path?

Starts with the definition of the variable and the variable is used in a computational or test predicate along a definition clear path.

What is full factorial classification in DOE?

Tests for each factor/value pair combination possible (like decision table)

What is validation?

Tests that show we are building the right program

What is the competent programmer hypothesis?

The belief that programmers generally only make minor errors

What is the coupling effect?

The belief that test data that can detect small errors can also detect complex errors.

What is the entry criteria for performance testing?

The system is reasonably stable, test environment is representative of customer site, have quantitative and measurable performance requirement, and have a resource and load generation tools

What is the goal of system level tests?

To find important problems and predict reliability

What is a way to reduce the number of tests created by cause and effect testing?

To make an assumption on the error handling

In mutation testing, the mutation score is a number between 0 and 1 that determines test quality

True

Where is control flow testing commonly used (what layer)?

Unit test

What layer(s) does structural/white box testing target?

Unit/Component Level

What layers of testing are there?

Unit/Component, Integration, System, Acceptance, Beta

What layer(s) do structural based testing apply?

Unit/Component, Integration, and System testing

50% of code defects from software modifications come from the new features but the other 50% comes from ___?

Untouched software

What perspective do black box tests target?

User

What are the two primary sources of information that help build black box tests?

User level documentation and requirement specification documentation

In a typical program the graph complexity equals?

V(G)=# of test predicates + 1

What is functional testing?

Verification and Validation, applies to all layers of testing

When does metamorphic testing help the most?

When the ground truth results are difficult to calculate (e.g. in graphics, big data, ML)

When do you stop testing?

When you've met your test objectives

What is model based regression testing ?

Where the tester takes the previous model and the model with modifications to it and looks for the differences in state and transitions

What are the steps of a model based test?

1. Create system model 2. Select test generation criteria (I.e. EP, BV) 3. Generate tests 4. Execute tests

What are the steps of state based testing?

1. Develop state testing tree 2. Identify test sequences (paths through tree) 3. Choose test cases to contain the sequence starting with the start state and ending with observable behavior. Must visit each state/transition combination

What is the steps of a pair wise/2-way classification?

1. Identify parameters that define each configuration 2. Partition each parameter (e.g. using EP, BV, etc) 3. Specify constraints prohibiting particular combinations of configuration partitions 4. Specify configurations to test which cover all partitions

What are the steps of multiple configuration testing?

1. Identify the parameters that define each configuration that could have an impact in the systems ability to meet functional and non-functional requirements 2. Partition groups (group similar parameters to reduce the number of configurations) 3. Identify configuration combinations (using boundaries, risk based, or DOE) 4. Execute

What are the steps of stress testing?

1. Identify the stress points 2. Develop strategy to stress the system at points identified in step 1 3. Verify the intended stress is actually generated (using resource monitor) 4. Observe behavior (and ensure functional correctness)

What x-way DOE design results in a reasonably well tested system?

4 to 6

What is asynchronous testing?

A specification/functional based testing where you create a timeline for each use case to test for potential events (e.g power failure)

What is boundary value testing?

A specification/functional based testing where you use equivalent partitions and test one less, one more, and on the edge of a petition. Like corner cases.

What is cause & effect testing?

A specification/functional testing where you can test multiple dependent variables. This can result in a lot of tests

What is generation fuzz testing?

A subset of fuzz testing where you create tests off of grammars/input format. Knowledge of the system would result in better test results.

What is stress testing?

A subset of performance testing where the goal is to verify the behavior of the system meets its requirements when the system's resources are saturated and pushed beyond their limits. This is a large amount of data/usage for a short amount of time.

What is volume testing?

A subset of performance testing where the goal is to verify the behavior of the system when the system is subjected to a large volume of activity of an extended period of time. Think about running the system 24/7.

What layer(s) do regression testing apply to?

All

What layers can be tested with black box testing?

All

What layers of testing can defect based testing be applied?

All

What is a test predicate?

An if/else or other conditional logic

What is mutation testing?

Combinatorial testing methodology where the tester makes syntactical modifications (e.g. flipping < and <=). If the tests pass with the mutated version and with the un-mutated version it means the tests aren't working.

What is exploratory session testing?

Combinatorial testing methodology where you adapt to what you find in the system while looking at a theme/tour. Not the same as adhoc testing

What is metamorphic testing?

Combinatorial testing where the tester works under the assumption that if one test results in the result x that if you modify the input you can see a correlation in that result y. AKA it is possible to predict changes to the outputs

What is defect based testing?

Combinatorial testing where the tests target know software vulnerabilities and errors (e.g. divide by 0). These are usually developed and evolved by previous defects.

What is fuzz testing?

Combinatorial testing where you provide the system with random/invalid/unexpected inputs. A test oracle is not needed because the system only monitors for unexpected errors/output or system crashes. Break it testing.

What are the three selective regression testing strategies?

Confidence Testing, Ripple Effect Analysis, and Code Analysis

What does load specification reflect?

Could be either different volumes of activity or different "mixes" of activity

What are the errors targeted in volume testing?

Counter overflow, memory leaks, resource depletion

What is statement coverage?

Create test cases such that every statement is executed (eg. x=5). Weakest form of coverage.

What are the testing types that help with the test oracle problem?

Fuzz testing and ??

What is Design if Experiments (DOE) in testing?

Is combinatorial testing methodology where you can systematically evaluate a system or process by either looking at a single factor or multiple factors. This is heavily utilized in manufacturing and quality engineering.

What is model based testing?

It is a specification/functional based testing where you develop a separate test model that adapts as the code's behavior/requirements change, it can lead to automated test generation, and improve precision and reduce ambiguity.

What is state based testing?

It is a specification/functional based testing where you use state diagrams to identify all paths possible.

What is Huang's theorem?

Let a,b,c be set of character sequences whose smallest string is at least one character long. Let T be a 2-character strong (anomaly). Then if T is a substring if a(b^n)c then T will appear a(b^2)c times. Basically means that the tester only needs to go through every possible path twice (and therefore limiting our search)

What are the challenges to mobile testing?

Many configurations, not necessarily company built and supportive (could be individual developer) and not have the resources (time or money) to properly test, touch screens/pressure/direction, resource constraints, may need emulators/simulators

What is modified condition decision coverage (MCDC)?

Most critical code gets tested more throughly

Is a test tree and a test matrix in cause and effect testing guaranteed to result in the same number of tests?

No

Does regression testing test code modifications?

No it tests the effect those modifications have on the system

Does DOE test all possible combinations of values of inputs?

No, DOE tests only pairs of values for each input, not all combinations

Is it true that testers primary goal of system testing is to find bugs.

No, the primary goal of system testing is to find important problems and predict reliability

What is fractional factorial classification?

Only a fraction of all combinations are tested. Orthogonal arrays often used to address limited combinations of factors.

What is selective regression testing?

Re-run a selected subset of tests based on the modifications made. This provides the tester with a standard confidence that irregardless of the modifications made to the system the previously developed code is still working as expected

What is full regression testing?

Re-run all existing tests in response to a code change. Normally very impractical.

What is code analysis in selective regression testing?

Re-run tests based on code deltas (I.e. for merge diff comparisons) and code coverage data

What is confidence testing in selective regression testing?

Re-run the same subset of tests regardless of what code was modified. Usually tests the most important features in the program.

What is the color pattern for test driven development?

Red, Green, Refactor

What is ripple effect analysis in selective regression testing?

Requires developers to identify the impact of changes made to system on other requirements or features

What are the control flow coverage layers from weakest to strongest?

Statement, Decision, Decision/Condition, Multiple Condition

Hat is the difference between stress and volume testing?

Stress testing is a large amount of data/pushing the system to its limits for a short period of time. Volume testing is a large amount of usage over an extended period of time.

What is static testing?

Structural based testing where code execution is not required where the tester models the flow of data in programs looking for data flow anomalies.

What is dynamic testing?

Structural based testing where the program is executed

What is mutation fuzz testing?

Subset of fuzz testing where the tester provides a "seed" (valid test data) and the fizzier creates tests off of the seed to create invalid tests.

What is symbolic execution?

Technique in static analysis for formally characterizing a path domain identifying a path condition. This must be expressed in terms of the initial values (a0, b0, etc. just means to simplify as much as possible). Goal format example [( ... AND ...) intersection (... OR ...) ]

What is non-functional testing?

Testing security requirements, performance, etc

What is the goal of exploratory testing?

The goal is to find defects that isn't scripted or automated

What does the number of basis paths mean for testing?

The number of basis paths is the minimum # of paths needed to build test cases and linear combinations for every other possible path

What is a definition clear path?

The path from node i where a variable x is defined and either used in a test predicate or computation node j where c is not redefined between nodes i and j.

What does it mean when the graph complexity (v(G)) is big?

The program has a higher likelihood of high risk, reliability issues, increased time to modify code, and an increase in #of defects

Do customers do there one testing for the acceptance level?

Yes

Is DOE effective in multiple configuration testing?

Yes, DOE is extremely beneficial in identifying the configuration combinations

Can symbolic execution find infeasible paths?

Yes, symbolic executions can prove that a path through the tree is infeasible.

Does defect based testing feed pay failures into the testing process?

Yes, this gets fed back into the "Defect Taxonomy" step of the process.