CSE 565
What is the process to find the basis paths in structured dynamic testing?
1. Select arbitrary path through graph 2. Flip first decision while keeping the test the same 3. Reset the first decision and flip the second 4. Continue until all decisions have been flipped
What is regression testing?
A non-functional/software quality testing methodology where it ensures previously developed and tested functions continue to work as specified after software modifications have been made
What is Configuration Testing?
A non-functional/software quality testing methodology where the tests focus on ensuring compatibility between required configurations (I.e. different OS, software versions, hardware, etc).
What is performance testing?
A non-functional:software quality testing methodology where the tester varies the load and stress on the system and sees how the system reacra
On waterfall development, when does testing start?
After all of the coding is complete.
What are the taxonomy categories called commonly used in defect based testing?
Beizer Generic Defect Taxonomy Categories
Does configuration testing test functional requirements, non-functional requirements, or both?
Both
Is black box behavioral/scenario based testing a validation, verification, or both activity?
Both
How is functional coverage demonstrated?
By mapping equivalence partitions to test cases
What is decision coverage?
Develop test cases such that all branches/test predicates get tested at least once. Decision coverage implies statement coverage.
What is multiple condition coverage?
Develop test cases such that all combinations of conditions in a decision are tested. This is the goal but difficult and at times impossible. This implies statement, decision, and decision/condition coverage.
What is decision/condition coverage?
Develop test cases such that each condition in a decision takes on all possible outcomes at least once and each decision takes on all possible outcomes at least once. Implies statement and decision coverage.
Why do defects cluster?
Due to complex code, programmer skill, etc.
What are some of the reasons why modifying existing software is such a high risk?
Error fixes, incorporation of new functions, code ripple effects, unintended feature interactions, changes in performance, resource sharing, etc
In agile development how often does testing happen?
Every day
What is a defect?
Fault in code made by an error
Is a 4-way DOE design full factorial or fractional factorial?
Fractional factorial
What are some of the drawbacks of pairwise/2-way tests?
May leave many functions untested and many software functions contain many parameters and factors
What is Orthagonal Defect Classification (ODC)?
Measurement method that uses the defect stream to provide measurability into the product and person.
What is the goal of DOE?
Minimize the number of tests we need to run.
What is an error?
Mistake made by human
Is black box static/requirement driven testing a validation, verification, or both activity?
Primarily verification
What is verification?
Showing we are building the program right/based on the requirements
What is a failure?
Software doesn't do what it's intended/supposed to. Caused by a defect.
What kind of testing is black box behavioral/scenario testing?
Specification/Functional Based Testing
What is Equivalence Partitioning (EP)?
Specification/Functional where you test different input classes
What is definition use path?
Starts with the definition of the variable and the variable is used in a computational or test predicate along a definition clear path.
What is full factorial classification in DOE?
Tests for each factor/value pair combination possible (like decision table)
What is validation?
Tests that show we are building the right program
What is the competent programmer hypothesis?
The belief that programmers generally only make minor errors
What is the coupling effect?
The belief that test data that can detect small errors can also detect complex errors.
What is the entry criteria for performance testing?
The system is reasonably stable, test environment is representative of customer site, have quantitative and measurable performance requirement, and have a resource and load generation tools
What is the goal of system level tests?
To find important problems and predict reliability
What is a way to reduce the number of tests created by cause and effect testing?
To make an assumption on the error handling
In mutation testing, the mutation score is a number between 0 and 1 that determines test quality
True
Where is control flow testing commonly used (what layer)?
Unit test
What layer(s) does structural/white box testing target?
Unit/Component Level
What layers of testing are there?
Unit/Component, Integration, System, Acceptance, Beta
What layer(s) do structural based testing apply?
Unit/Component, Integration, and System testing
50% of code defects from software modifications come from the new features but the other 50% comes from ___?
Untouched software
What perspective do black box tests target?
User
What are the two primary sources of information that help build black box tests?
User level documentation and requirement specification documentation
In a typical program the graph complexity equals?
V(G)=# of test predicates + 1
What is functional testing?
Verification and Validation, applies to all layers of testing
When does metamorphic testing help the most?
When the ground truth results are difficult to calculate (e.g. in graphics, big data, ML)
When do you stop testing?
When you've met your test objectives
What is model based regression testing ?
Where the tester takes the previous model and the model with modifications to it and looks for the differences in state and transitions
What are the steps of a model based test?
1. Create system model 2. Select test generation criteria (I.e. EP, BV) 3. Generate tests 4. Execute tests
What are the steps of state based testing?
1. Develop state testing tree 2. Identify test sequences (paths through tree) 3. Choose test cases to contain the sequence starting with the start state and ending with observable behavior. Must visit each state/transition combination
What is the steps of a pair wise/2-way classification?
1. Identify parameters that define each configuration 2. Partition each parameter (e.g. using EP, BV, etc) 3. Specify constraints prohibiting particular combinations of configuration partitions 4. Specify configurations to test which cover all partitions
What are the steps of multiple configuration testing?
1. Identify the parameters that define each configuration that could have an impact in the systems ability to meet functional and non-functional requirements 2. Partition groups (group similar parameters to reduce the number of configurations) 3. Identify configuration combinations (using boundaries, risk based, or DOE) 4. Execute
What are the steps of stress testing?
1. Identify the stress points 2. Develop strategy to stress the system at points identified in step 1 3. Verify the intended stress is actually generated (using resource monitor) 4. Observe behavior (and ensure functional correctness)
What x-way DOE design results in a reasonably well tested system?
4 to 6
What is asynchronous testing?
A specification/functional based testing where you create a timeline for each use case to test for potential events (e.g power failure)
What is boundary value testing?
A specification/functional based testing where you use equivalent partitions and test one less, one more, and on the edge of a petition. Like corner cases.
What is cause & effect testing?
A specification/functional testing where you can test multiple dependent variables. This can result in a lot of tests
What is generation fuzz testing?
A subset of fuzz testing where you create tests off of grammars/input format. Knowledge of the system would result in better test results.
What is stress testing?
A subset of performance testing where the goal is to verify the behavior of the system meets its requirements when the system's resources are saturated and pushed beyond their limits. This is a large amount of data/usage for a short amount of time.
What is volume testing?
A subset of performance testing where the goal is to verify the behavior of the system when the system is subjected to a large volume of activity of an extended period of time. Think about running the system 24/7.
What layer(s) do regression testing apply to?
All
What layers can be tested with black box testing?
All
What layers of testing can defect based testing be applied?
All
What is a test predicate?
An if/else or other conditional logic
What is mutation testing?
Combinatorial testing methodology where the tester makes syntactical modifications (e.g. flipping < and <=). If the tests pass with the mutated version and with the un-mutated version it means the tests aren't working.
What is exploratory session testing?
Combinatorial testing methodology where you adapt to what you find in the system while looking at a theme/tour. Not the same as adhoc testing
What is metamorphic testing?
Combinatorial testing where the tester works under the assumption that if one test results in the result x that if you modify the input you can see a correlation in that result y. AKA it is possible to predict changes to the outputs
What is defect based testing?
Combinatorial testing where the tests target know software vulnerabilities and errors (e.g. divide by 0). These are usually developed and evolved by previous defects.
What is fuzz testing?
Combinatorial testing where you provide the system with random/invalid/unexpected inputs. A test oracle is not needed because the system only monitors for unexpected errors/output or system crashes. Break it testing.
What are the three selective regression testing strategies?
Confidence Testing, Ripple Effect Analysis, and Code Analysis
What does load specification reflect?
Could be either different volumes of activity or different "mixes" of activity
What are the errors targeted in volume testing?
Counter overflow, memory leaks, resource depletion
What is statement coverage?
Create test cases such that every statement is executed (eg. x=5). Weakest form of coverage.
What are the testing types that help with the test oracle problem?
Fuzz testing and ??
What is Design if Experiments (DOE) in testing?
Is combinatorial testing methodology where you can systematically evaluate a system or process by either looking at a single factor or multiple factors. This is heavily utilized in manufacturing and quality engineering.
What is model based testing?
It is a specification/functional based testing where you develop a separate test model that adapts as the code's behavior/requirements change, it can lead to automated test generation, and improve precision and reduce ambiguity.
What is state based testing?
It is a specification/functional based testing where you use state diagrams to identify all paths possible.
What is Huang's theorem?
Let a,b,c be set of character sequences whose smallest string is at least one character long. Let T be a 2-character strong (anomaly). Then if T is a substring if a(b^n)c then T will appear a(b^2)c times. Basically means that the tester only needs to go through every possible path twice (and therefore limiting our search)
What are the challenges to mobile testing?
Many configurations, not necessarily company built and supportive (could be individual developer) and not have the resources (time or money) to properly test, touch screens/pressure/direction, resource constraints, may need emulators/simulators
What is modified condition decision coverage (MCDC)?
Most critical code gets tested more throughly
Is a test tree and a test matrix in cause and effect testing guaranteed to result in the same number of tests?
No
Does regression testing test code modifications?
No it tests the effect those modifications have on the system
Does DOE test all possible combinations of values of inputs?
No, DOE tests only pairs of values for each input, not all combinations
Is it true that testers primary goal of system testing is to find bugs.
No, the primary goal of system testing is to find important problems and predict reliability
What is fractional factorial classification?
Only a fraction of all combinations are tested. Orthogonal arrays often used to address limited combinations of factors.
What is selective regression testing?
Re-run a selected subset of tests based on the modifications made. This provides the tester with a standard confidence that irregardless of the modifications made to the system the previously developed code is still working as expected
What is full regression testing?
Re-run all existing tests in response to a code change. Normally very impractical.
What is code analysis in selective regression testing?
Re-run tests based on code deltas (I.e. for merge diff comparisons) and code coverage data
What is confidence testing in selective regression testing?
Re-run the same subset of tests regardless of what code was modified. Usually tests the most important features in the program.
What is the color pattern for test driven development?
Red, Green, Refactor
What is ripple effect analysis in selective regression testing?
Requires developers to identify the impact of changes made to system on other requirements or features
What are the control flow coverage layers from weakest to strongest?
Statement, Decision, Decision/Condition, Multiple Condition
Hat is the difference between stress and volume testing?
Stress testing is a large amount of data/pushing the system to its limits for a short period of time. Volume testing is a large amount of usage over an extended period of time.
What is static testing?
Structural based testing where code execution is not required where the tester models the flow of data in programs looking for data flow anomalies.
What is dynamic testing?
Structural based testing where the program is executed
What is mutation fuzz testing?
Subset of fuzz testing where the tester provides a "seed" (valid test data) and the fizzier creates tests off of the seed to create invalid tests.
What is symbolic execution?
Technique in static analysis for formally characterizing a path domain identifying a path condition. This must be expressed in terms of the initial values (a0, b0, etc. just means to simplify as much as possible). Goal format example [( ... AND ...) intersection (... OR ...) ]
What is non-functional testing?
Testing security requirements, performance, etc
What is the goal of exploratory testing?
The goal is to find defects that isn't scripted or automated
What does the number of basis paths mean for testing?
The number of basis paths is the minimum # of paths needed to build test cases and linear combinations for every other possible path
What is a definition clear path?
The path from node i where a variable x is defined and either used in a test predicate or computation node j where c is not redefined between nodes i and j.
What does it mean when the graph complexity (v(G)) is big?
The program has a higher likelihood of high risk, reliability issues, increased time to modify code, and an increase in #of defects
Do customers do there one testing for the acceptance level?
Yes
Is DOE effective in multiple configuration testing?
Yes, DOE is extremely beneficial in identifying the configuration combinations
Can symbolic execution find infeasible paths?
Yes, symbolic executions can prove that a path through the tree is infeasible.
Does defect based testing feed pay failures into the testing process?
Yes, this gets fed back into the "Defect Taxonomy" step of the process.