Midterm Review
Strong equivalence class testing
(Multi dimensional), based on the cartesian product of the partition subsets (AxBxC), i.e. testing all interactions of all equivalence classes. # cases = |A| x |B| x |C|
White box testing
A procedure to derive and/or select test cases based on an analysis of the internal structure of a component or system. Aka structural, open box testing - explicit knowledge of internal working of SUT is used to select, execute, and collect test data - using specific knowledge of source code to define tests
Path
A sequence of nodes. Each pair of nodes is an edge
Test suites
A set of tests that all share the same fixture. The order of the tests shouldn't matter
NIST's covering array tool
A solution to the issue of finding the minimum covering array. Defines variables in SUT and add constraints among variables. Proposed covering array can be output as a spreadsheet
Doubles
A test double is a replacement for a DOC (Dependent on component), e.g. sending an email if a request fails
Test result formatter
A test runner produces results in 1 or more output formats
Maintainability
Ability to undergo repairs and modifications. A measure of the time to service restoration since the last failure occurrence, or measure of the continuous delivery of incorrect service
Non-regression verification
After correction, repeat the verification process in order
CFG
Control flow graph/flowchart A directed graph with each node corresponding to a program statement. Each directed edge in the set indicates flow control from 1 statement of program to another
3 attributes of software structures
Control flow structure- sequence of execution of instructions of the program Data flow- keep track of data as it's created and handled by the program Data structure- the organization of data itself independent of the program
DFG
Data flow graph
Test automation
Embed test values into executable scripts
test coverage tools
Emma, CodeCover, JaCoCo
Scope
Ensure that each unit has been implemented correctly (looking for faults in isolated areas- class/obj)
Completeness
Entire input set is covered
Node (line or statement) coverage
Execute every program statement
What is observable? (fault, failure, error?)
Failure
Failure Density
Failure per kilo-lines of developed code (or per FP), e.g. 1 failure per KLOC, 0.2 failures per FP, etc.
Mock
Fake java class that replaces the depended class and can be examined after the test is finished for its interactions with the class under test, ex. ask it whether a method was called or how many times it was called. Typical mocks are classes with side effects that need to be examined
Test drives
Modules that act as temporary replacements for a calling module and give the same output as that of the actual product
Reachability
Not all statements are always reachable. We can't always decide if a statement is reachable and how many are. When one doesn't reach 100%, it's hard to see why
Requirement analysis
Studying requirements from a testing perspective to identify the testable requirements. Results in a Requirement Traceability Matrix (RTM)
SUT
System Under Test
Software Testing
Techniques to execute programs with the intent of finding as many defects as possible and/or gaining sufficient confidence in the software system under test
Functional testing
Testing the functional requirements. Checking the correct functionality of a system
Fault prevention
To avoid fault occurrences by construction. Attained by quality control techniques employed during the design and manufacturing of software. It intends to prevent operational physical faults. Example techniques: design review, modularization, consistency checking, structured programming, etc. Activities: requirement review, design review, clear code, establishing standards, using CASE tools with built-in check mechanisms
Disjoint classes
To avoid redundancy so only 1 element of each equivalence class is tested
Category-partition testing
combines BVA and EC with domain expertise. Constraints can also be defined
Def-pair set
du(ni, nj, v): all simple paths with respect to a given variable from a given definition (ni) to a given use (nj)
Def-path set
du(ni, v): all simple paths with respect to a given variable defined in a given node
Prime flow graph
flow graphs that can't be decomposed non-trivially by sequencing and nesting
Steps for unit testing
get code to test (SUT) write program that calls SUT and executes it check if results meet expected
Loops
looping structures (for, while, etc.)
Weak equivalence class testing
(1 dimensional), choosing 1 variable from each equivalence class so that all classes are covered
Fault
(bug) A cause for either a failure of the program or an internal error (e.g. an incorrect state, incorrect timing). It must be detected and removed. - They can be removed without execution (code inspection, design reviews). - Removal of these due to execution depends on the occurrence of the associated failure. - Occurrence depends on the length of execution time and operational profile
Exploratory test case design
(human-based), designing test values based on domain knowledge of the program and human knowledge of testing, exploratory testing
Characteristics of unit testing
- unit test must specify only 1 unit of functionality - must be fast to build - doesn't access DB or file system - doesn't communicate via network - doesn't require any special environment setup - leaves environment in the same state as before - focus on developed components only
Error
1. A human action that results in software combining a fault 2. A discrepancy between a computed, observed, or measured value/condition, and the true, specified or theoretically correct value/condition
Trivial partitions
1. A set containing all expected or legal inputs 2. a set containing unexpected or illegal inputs These can be divided into subsets on which the application is required to behave differently
Means of Dependability
1. Fault prevention- how to prevent occurrence/introduction of faults 2. Fault tolerance- how to deliver correct service in the presence of faults 3. Fault removal- how to reduce the number or severity of faults 4. Fault forecasting- how to estimate the present number, the future incidence, and the likely consequences of faults
3 key points of reliability
1. Reliability depends on how the software is used, thus a model of usage is required 2. Reliability can be improved over time if certain bugs are fixed (reliability growth), thus a trend model (aggregation or regression) is needed 3. Failures may happen at random times, thus a probabilistic model of failure is needed
How to build equivalence classes
1. don't forget equivalence classes for invalid inputs 2. look for extreme range of values 3. look for max size of memory (stackable) variables 4. if an input must belong to a group, 1 equivalence class must include all members of the group 5. analyze levels for boolean variables 6. look for dependent variables and replace them with their equivalents
Control flow coverage
1. from source code, create a CFG 2. design test cases to cover certain elements of CFG 3. decide upon appropriate coverage metrics to report test results (statement, decision, condition, path coverage metrics) 4. execute tests, collect and report coverage data
Stub
A fake class that comes with preprogrammed return values. Injected into the class under test to give control over what's being tested as input, ex. a database connection that allows you to mimic a connection scenario without a real database. A simple test double that supplies responds to requests from the SUT. Contains pre-defined responses to specific requests. Often hand-coded
Mock object
A fake object that decides whether a unit test has passed or failed by watching interactions between objects. Needed when a unit of code depends on an external object. A dummy implementation for a class in Mock framework
Fault tolerance
A fault-tolerant system is capable of providing specified services in the presence of a bounded number of failures. The use of techniques to enable continuous delivery of services during system operation. It's generally implemented by error detection and subsequent system recovery. Based on the principle: "act during operation defined during specification and design" Process: 1. detection of faults and causes, 2. assessment to which the system state has been damaged or corrupted, 3. recovery (remaining operational), 4. fault treatment and continued service (locate and repair fault to prevent it from happening again)
Reliability
A measure of the continuous delivery of correct service. The probability that a system or a capability of a system functions without failure for a specified time or number of natural units in a specified environment, given that the system was functioning properly at the start of the time period. The probability of a failure free operation for a specified time in a specified environment for a given purpose The most important attribute of software
Availability
A measure of the delivery of correct service with respect to the alteration of correct and incorrect service. Avail = uptime/(uptime + downtime) Avail = MTTF/(MTTF + MTTR) = MTTF/MTBF
Test path
A path that starts at an initial node and ends at a final node. Represents execution of test cases
Test automation framework
A set of assumptions and tools that support test automation
Profile based testing
A system may have several functionalities but only a subset of them need to work together at any time. The usage of various operations may vary. Write tests for operations based on their frequency of usage
Service
A system's behaviour as it is perceived by users
Black box testing
A testing approach that focuses on the functionality of the application or product and does not require knowledge of the code. Applies at all granularity levels. Aka specification-based testing.
Service Restoration
A transition from incorrect to correct service
All definitions coverage
ADC. uses every definition. For each set of du-paths, the test requirement contains at least 1 path in the set. For each definition, at least 1 use must be reached
All du-path coverage
ADUPC. follow all du-paths. For each set of du-paths, test requirement contains every path in the set. For each def-use pair, all paths between def and use must be covered
All uses coverage
AUC. get to every use. For each set of du-paths, test requirement contains at least 1 path in the set. All uses must be reached for each definition
Safety
Absence of catastrophic consequences on users and environment. An extension of reliability: safety is reliability with respect to catastrophic failures. When the state of correct and incorrect service due to non-catastrophic failures are grouped into a safe state (free from catastrophic damage, not from danger), safety is a measure of continuous safeness, or the time to catastrophic failure
Integrity
Absence of improper system state alterations
Confidentiality
Absence of unauthorized disclosure of information
Qualitative Evaluation
Aims to evaluate the extent to which some of the attributes of dependability are satisfied; those attributes are then viewed as measures of dependability
Quantitate Evaluation (ordinal)
Aims to identify, classify, rank the failure modes or event combinations that lead to system failures
Testing
Allows for defect detection and starts later in the life cycle
Crowd testing
An approach to testing in which testing is distributed to a large group of testers.
Failure
An event that occurs when the delivered service deviates from the correct service. It is thus a transition from correct to incorrect service, i.e. to not implementing the system function. Any departure of system behaviour in execution from user needs. Caused by a fault, which is usually caused by human error
Test runner
An executable program that runs tests implemented using JUnit framework and reports test results
User
Another system (physical, human) that interacts with the system at the service interface
Data flow coverage
Augment the CFG with extra information - branch predicates - defs: statements that assign values to variables - uses: statements that use variables
Dependability attributes
Availability- readiness for correct service Reliability- continuity of correct service Safety- absence of catastrophic consequences on users and environment Confidentiality- absence of unauthorized disclosure of info Integrity- absence of improper system state alterations Maintainability- ability to undergo repairs and modifications Availability is always required, and others may/not be.
BVT
Boundary Value Testing. Mistakes are made in processing values at/near the boundaries of equivalence classes
Validation
Checking whether the system meets customer's actual needs (building the right product)
Subsumption
Coverage criterion C1 subsumes C2 iff every test set that satisfies C1 also satisfies C2, e.g. branch coverage subsumes statement coverage. - 100% branch coverage is 100% statement coverage
Data flow graph
DFG. A variation of CFG where nodes are annotated by "def" and "use" which is broken into p-use and c-use. arcs are annotated by predicate conditions as true or false. A def-use table can help identify the test paths
Structural coverage criteria
Defined on a graph just in terms of nodes and edges
Correct Service
Delivered when the service implements the system function
System Outage
Delivery of incorrect service
Condition coverage
Design a test set so that each individual condition in the program is both true and false. May cause branch coverage to suffer. Should usually consider both branch and condition coverage together by executing the program for each element in the set **Make individual conditions true and false
Modified Condition Decision Coverage (MCDC)
Effectively test important combinations of conditions. Only combinations of values so that every atomic condition impacts the overall condition's truth value, i.e. the outcome of a decision changes as a result of changing each single condition
ECT
Equivalence Class Testing 1. Decide how many independent variables you have 2. for each, how many distinct partitions for each variable we have 3. select number of test cases based on weak, strong, or robustness criteria 4. create test cases using acceptable values assigned from each distinct partition for each variable involved 5. review test cases: remove redundant, add tests for problems perceived, etc. 6. repeat steps until satisfied with test cases
Test evaluation
Evaluate results of testing, report to developers
Statement coverage
Faults can't be discovered if code containing them are not executed. Equivalent to covering all nodes in CFG. Several inputs execute the same statements. Using this type of coverage may lead to incompleteness
Manual scripted testing
Follows a path written by tester that includes test cases and documented steps. No deviation is laid out in the script
Method
For a given class, create another of the same name with "Test" appended to test it, which contains various test cases to run. Each method looks for particular results and either passes/fails
Testing through SDLC
Identifying test cases early to help shorten the development time.
Testing in iterative model
Iterative process starts with implementing a subset of the requirements. At each iteration, design modifications are made and new functional capabilities are added. Idea is to develop a system through repeated cycles and in incremental portions
JUnit
Java testing framework used to write and run tests. Open sourced
Combinatorial testing
May be impractical to test all possible combinations of values for all variables. In this type of testing, we combine values systematically but not exhaustively. We assume that unplanned interactions will happen among a few parameters. If all faults triggered by interaction of t or less variables, then testing all t-way combinations can provide strong assurance
Procedure nodes
Nodes with out degree = 1
Predicate nodes
Nodes with out degree other than 1 or 0 (branches)
How to build a CFG
Nodes: - statement nodes represent single entry single exit sequence statements - predicate nodes represent conditions for branching - auxiliary nodes for completing the graph
Equivalence class
Partitions of the input space in such a way that input data have the same effect on the SUT
Fault removal
Performed in both the development phase and operational life of a system. During the development phase, there are 3 steps: verification, diagnosis, and correction
Defect
Refers to either fault (cause) or failure (effect)
Decision/branch coverage
Relates to decisions in a program. Given a CFG, select a test such that by executing the program for each test case in the set, each edge of CFG's decision nodes is traversed at least once. Need to exercise all decisions that traverse the control flow of the program with T or F values **Make compound predicate true and false
RTM
Requirements traceability matrix. Lists the test cases as rows and requirements as columns. An "X" is placed in each box that for a test case that fills that requirement. Helps engineers to see whether all requirements are tested and helps project managers to see if they are on schedule with development and testing
Test execution
Run tests on the software and record the results
Decision table
Sections: - conditions: lists conditions and their combinations, express relationships among decision variables - actions: lists responses to be produced when corresponding combinations of conditions are true 1 test case for each rule from this table cases = 2^conditions
Path coverage
Select a test set such that by executing the program for each test case, all paths leading from the initial to final node of program''s CFG are traversed. Counts the number of full paths from input to output through a program that gets executed. Full path coverage leads to full branch coverage
SDLC
Software Development Life Cycle
STLC
Software Testing Life Cycle
Inspection
Strict, close examinations conducted on specifications, design, code, tests, etc. It allows for defect detection, prevention, isolation. Starts early in the life cycle. Up to 20x more efficient than testing. Code reading detects 2x as many defects/hour as testing does. 80% of coding errors are found by this method.
TDD
Test Driven Development. Test cases are the requirements of the system. Shortens the programming feedback loop, guarantees development of high quality software. Test writing is very time consuming though
Exploratory testing
Test cases are designed an executed at the same time. About enabling choice but not constraining it Human tester using brain to create realistic scenarios that will either cause software to fail or succeed. No formal plan. All about varying things
Coverage metrics
Test coverage measures the amount of testing performed by some tests. Coverage = # cvg items exercised/total # of cvg items x100%
Exhaustive testing
Testing a software system using all possible inputs. Usually impossible
Criteria-based testing
Testing based on a set of criteria
Manual regression testing
Testing done to check that a system update doesn't re-introduce faults that have been corrected earlier. Usually performed after a bug fix code is checked into the code repo (smoke testing)
Non-functional testing
Testing the non-functional requirements: performance, localization/internationalization, recovery, security, portability, compatibility, usability, scalability, reliability, etc.
Scripted testing
Tests are first designed and recorded then may be executed at some later time or by the same/different tester. Misses the same things every time. In small (functions), focuses on a simple functionality and excludes everything else. Easy to localize any faults In large (operation), focuses on a set or sequence of functionalities. Difficult to localize faults
Failure Effect
The consequences of a failure mode on an operation, function, status of a system, process, activity, or environment. The undesirable outcome of a fault of a system element in a particular mode.
Test execution
The execution of an individual unit test: @Before, @Test, @After
Failure Mode
The manner in which a fault occurs, i.e. the way in which the element faults
Cyclomatic complexity
The number of regions in a flow graph. A program's complexity can be measured by the cyclomatic number of the program flow graph. The cyclomatic number can be calculated by: - flow graphs - code - number of regions in flow graph
Failure Intensity (Failure rate)
The rate that failure occur, i.e. the number of failures per natural or time unit. A way of expressing system reliability, e.g. 5 failures per hour
Test fixtures
The set of preconditions or state needed to run a test (context)
Test fixture
The state of a test - objects and variables that are used by more than 1 test - initializations - reset values
Factors affecting software quality
Time Cost Quality/Reliability
TBF
Time between failure
Exceptions
Uses @Test(expected = Exception.class) and make sure the function throws that exception
String-normal ECT
What is this an example of, weak or strong ECT? A has 3 partitions, B has 4, C has 2. 3x4x2 = 24 test cases
Weak-normal ECT
What is this an example of, weak or strong ECT? A has 3 partitions, B has 4, C has 2. max(3,4,2) = 4 test cases
Function
What the system is intended to do, described by the functional specification
Test coverage
Whenever we can count things and tell whether each of those things has been tested by some test
Boundary value analysis
While partitioning, select tests from inside equivalence classes. Focuses on tests at/near boundaries of equivalence class - at min, just below, just above - nominal value - at max, just below, just above
WCT
Worst case testing. When more than 1 variable uses an extreme value
Assertion
a line of code that compares 2 given values (one from test case, one from oracle) Verifies the behaviour/state of the unit under test, e.g. true, null, false
Def
a location where a value for a variable is stored
use
a location where a variable's value is accessed (read, evaluated, computed)
Du-pair
a pair of definition and use for a variable
Covering array
a set of concrete test cases that covers all t-way interactions among the k factors with all combinations of l levels of values so that the combination appears in at least 1 test
Profile
a set of disjoint alternatives called elements that represent that phenomenon together with their occurrence probabilities
Test case
a set of inputs and the expected outputs (oracle) for a unit/module/system under test. Specifying a function's direct input variables and the expected outcome. Specifying the indirect input variables gives a test case the necessary context.
Test suite
a set of test cases
Test Suite
a set of test cases that needs to be run together with the SUT
Du-path
a simple path where the initial node of the path is the only defining node of variable in the path - du(ni, v): set of du-paths that start at ni for variable v - du(ni, nj, v): the set of du-paths from ni to nj for variable v
Subpath
a subsequence of nodes in p is a subpath of p
Definition occurrence
a value is written (bound) to a variable
Direct input variable
a variable that controls the operation directly. Ex. arguments, selection menu, entered data field. Important during exploratory/unit test
Indirect input variable
a variable that only influences the operations or its effects are propagated to the operation. Ex. traffic load, environment variables. Important during integration testing.
Parameterized tests
allow running the same test repeatedly using different input test data. 5 steps: 1. annotate test class with @RunWith 2. create a public static method annotated with @Parameters that returns a collection of objects (Array) as test data set 3. create a public constructor that takes in 1 row of test data 4. create an instance variable for each column of test data 5. create your test cases using the instance variables as the source of the test data
Robustness testing
below lower bound and above upper bound are included in the test cases
Computational use
c-use: compute a value for defining other variables or output values
Tabular representation
composed of a list of operation names and their probability of occurrence
def(n)
contains variables that are defined at node n (written)
use(n)
contains variables that are used at node n (read)
c-use pairs
dcu(x). The set of all nodes using variable x from the live definitions of the variables at a given previous node i
Boundary conditions
defining the edge cases of the equivalent classes
Criteria based test case design
design test values to satisfy coverage criteria or other engineering goals
p-use pairs
dpu(x). The set of all edges such that there is a def-clear path from node i to edge, and x is used at node k
Edge (decision or branch) coverage
execute every branch
Testing approaches
exhaustive testing random testing (ad-hoc, exploratory) partitioning
Oracle
getting a value from someone/something where we know it's correct
Equivalent classes
if a variable is changing, are they equivalent with respect to the functionality being tested or not. A group of test cases are equivalent if: - they all test the same unit - if 1 test case can catch a bug, the others probably will too - if 1 test case doesn't catch a bug, the others probably won't either - all involve same input variables - all affect same output variables
reach
if there's a def-clear path from nodes m to p with respect to v1, then the definition of v at m reaches the use at p
Mock frameworks
jMock, Mockito, EasyMock Provide auto generation of mock objects that implement a given interface, methods for declaring and asserting your expectations, and logging of what calls are performed on the mock objects
parameters that make a good test
long, descriptive names timeout one test per method avoid logic
Start node
nodes with in degree 0
Terminal (end) nodes
nodes with out degree 0
Def-clear path
p with respect to v is a sub-path where v is not defined at any nodes in p Any path starting from a node at which a variable is defined and ends at a node where that variable is used without redefining the variable anywhere else on the path
Predicate use
p-use: a variable is used to decide whether a predicate evaluates to true/false
Fault forecasting
performing an evaluation of the system behaviour with respect to fault occurrence or activation. 2 aspects: qualitative/ordinal evaluation and quantitative/probabilistic evaluation
Test planning
preparing a test plan document for various testing, selecting tools for testing, test cost/effort estimation, resource planning and role determining. Deliverables are a test plan document and effort estimation document
Data flow coverage criteria
requires a graph to be annotated with references to variables
limitations of human based testing
scalability, bias scaling for complex systems repeatability
Operational profile
set of operations (names and frequency) and their probability of occurrences
Data flow based testing
test the connections between variable definitions (write) and variable uses (read). The starting point is a variation of a CFG annotated with the location of all variables
Acceptance testing
testing by users to check whether the system satisfies requirements (aka alpha testing)
system testing
testing the complete system before delivery
Integration testing
testing to expose problems arising from the combination of components
Infeasibility
the du-pair is not feasible
Length
the number of edges. A single node has a path of length 0
Verification
the process of checking whether a system adheres to given properties/verification conditions. If it doesn't, the other steps follow: diagnosing faults that prevented verification, and then performing necessary corrections (building the product right)
Multiple condition
using condition coverage on some compound condition C implies that each simple condition within C has been evaluated to true and false. However, it doesn't imply that all combinations of the values of the individual simple conditions in C have been exercised
Use occurrence
value of a variable is read (referred)