Software quality assurance 2
pdf - probability density function
# defects which will be found per life cycle phase
test effectiveness
(# defects removed by phase / # total # defects found in this hase or later phase)
Calculating DRE v2 - steps
1) take req% and place in ID box for D_req. ID * (1 - % of FE (given)) = PD. ID - PD = RD sum to get totals per row
3 critera to follow
1) use every def, 2) get to every use 3) follow all du paths
prime path
: A simple path that does not appear as a proper subpath of any other simple path
perfective maintenance
: Modification of a software product after delivery to improve performance or maintainability
phase containment effectiveness
= (# phase i errors) / (number of phase i errors + number of phase i defects)
simple path
A path from node ni to nj is simple if no node appears more than once, except possibly the first and last nodes are the same. no internal loops
preventative maintenance
Modification of a software product after delivery to detect and correct latent faults in the software product before they become effective faults.
def-clear
a path from i to j is def clear for var v if v is not given another value on any of the nodes or edges in the path
partitioning
a sampling strategy to simplify testing. subjective due to ec picking process
basic block
a sequence of statements that if the fisrt statement is executed, all statements will be (no brances)
du-path
a simple subpath that is def=clear with respect to v from a def of v to a use of v
tour
a test pat p tours subpath q if q is a subpath of p. side trips (edge in same order)and detours (node in same order)
galin extended cost model
adds managerial costs to each to prepare and to handle failure costs
sci verison software configuration item version
approved state of sci at any given time
sci - software configuration item
approved unit of software code for config mgmt and treated as distinct entity
Calculating DRE V1 - steps
assume 100% defects, get average % of defects originating in each phase, calculate cost of each plan via matrix
data flow coverage
augmenting the control flow graph
equivalence class partintioning
blackbox aimed to increase efficiency of testing and improve coverage
peer reviews
both inspections and code walkthrough (formal and less formal) review leader, author of review, and specialized professions. Feedback only, not formal sign off
limitation of metrics
budgetary constraints allocating resources, human factors, biased reporting, partial reporting
tools for maintance
bug repo, debugging tools, versioning tools, configuration mgmt, testing tools
timetable metrics
calculate milestones completed on time / total # of milestones
auto vs manual testing
can automate performing tests, prepping test log, regression tests. manual planning, design, and test setup
fuzzy boundaries
can't use mutual exclusive ranges since too many combinations. must use EC instead
tasks of config mgmt
control software change (proposed changes), release sci and versions, verify compliace with scm procedures. consider urgency of change, effect it will have on timetables and testing, cost of change
components of maintenance
corrective maintenances (bug fix), adaptive maintanance (changes for new os, etc), and functionality improvement maintenance (perfective and preventative maintenanace)
2 types of costs
cost of control (prevention and appriasal) cost of failure of control (internal failure costs, extrenal failure costs)
RDRC
cost of defect removal per phase. unit cost of removal
external failure costs
customer complaints, damages, reduction of sales to customers, increase in sales team cost to raise sale amount
structural coverage criteria
defined on a graph in terms of edges and nodes
test requirements
describe properties of test paths
software plan template
determine scope of tests, testing environemnt, test details, test schedule. test results, summary for totoal # errors, special events and proposals
testing process (4 steps)
determine strategy, plan tests, design tests, peform tests
all-du-path coverage
each and every paths between all defs and uses
availability
full availability, vital available, total unavailability (down time failure hours / hours software in service)
why config mgmt?
handling correct version for next development task, who can provide what tool or document, what changes have been made since A happened, where can i find who uses version 1.1 of our software
defect injection
if defects carry over from previous steps, they are injected into the next phase dre = defects removed at step / (defects existing + defects injected)
prevention cost
investingg in sqa prevention activities, infrastructure
kloc
kilolines of code. determine quality by lines of code
software evolution models
linear and tree branching and successive changes for releases
maintanance plan
list of contractual servies agreed upon. scope budget, schedule personnel
software quality metrics requreiements
must be relevant, valid, reliable, comprehensible, mutually exclusive
error density metrics
normalizes by dividing by kloc or wce or nfp (number of coding errors/kloc)
nce
number of coding errors
nfp
number of cuntion points - function point is a "unit of measurement" to express the amount of business functionality an information system (as a product) provides to a user. Function points are used to compute a functional size measurement (FSM) of software. The cost (in dollars or hours) of a single unit is calculated from past projects
DRE metrics
number of development errors / (number of errors + number of failures a year)
motorola
number of prelease defects / (number of prerelease defects + number of post release defects)
reliability
probabilyt of failure free operation over specified time
software quality metics
quantitative measure of degree that an item is of quality
test case sources
random sampling of real life cases and synthetic test cases (simulated)
metrics of reliability
rate of fault occurrence, prob of failure on demaind, MTTF, # transactions, # calendar dates, raw execution time
dynamic reliability modesl
rayleigh model, exponential model
internal failure cost
redesign due to test findings, repeating design reviews due to test findings
appriasal cost
reviwing, testing, getting external participants
version development policy
sequential version policy tree version policy
process metrics categories
software quality metrics, software proces time table metrics, DRE metrics, productivity metrics
software config mgmt
sqa component for applying tools and procedures that allow completion of tasks required to maintain sci and software configuration versions
baseline version
stable and relsead version that was tested thoroughly
types of reliability modesl
static (uses product characteristics to estimate # defects) and dynamic (uses statistical distribution to estimate future reliability)
3 types of external participants
subcontractors, 3rd party provider of modules, customers
path(t)
test path is executed by test t
cdf - cumulative density function
total # defects will be found as a function of life cycle phase
domain testing
two paths are equivalent if the program would take same path in response to each
equivalence
two test cases are equivalent if you expect the same result for each
testing process and risk
use risk to prioritize test cases ranking. quality plans need to include test plans (shrine)
helpdesk quality metrics
uses hd calls to calculate. # calls/ year. corrective code required in maintenance, # fp to be maintained
productivity metrics
uses working hours. poor metric
du paths
using du pairs, get paths that are unique
release docs
version #, revision #, date, list of installations, list of sci in version, installation instructions, changes in new version, further development process
raylegh model
weibull model with a m value of 2. formal paramtric model to produce estimate of future defect rate
wce
weight of coding errors (coding errors x relative weight)
error severity metrics
weighted code errors / number of code errors to get average severity of code errors (uses weights provided to compare on severity)
defect origin
where defects are first introduced
equivalence class
- set of input variable values that produce the same output results or are processed identically.
bottom up testing
REquires Test Driver. test all lower levels, the combine and test one level higher
top down testing
Requires Stubs. test higher functionality with stubs at the lowest level. then implement downwards.
Phases of software development RDU ID SO
Requirmenents, design, unit coding, integration, documentation, system testing,operation
types of testing
alpha (end users at dev's site), beta (end users in their environment) , uat (testing for sign off), performance testing (load and stress), security testing, UX testing, unit, integration, subsystem, system testing
prime path coverage
determine all simple paths (no duplicate nodes except first and last), then remove all duplicates or subpaths
What is the inverted v diagram
dev activity - requirements, system/HLD, architecture design, module design Coding unit testing, integration testing, system testing, UAT
boehm and ross's top 10 software risk items
developing wrong functions, unrealistic schedule and budget, wrong interface for user, gold plating, continuous stream of req changes, personnel shortfalls
validation process
does product satisfy specified requirements
verification process
does the product satisfy conditions at the start of the phase
all use coverage
every def reaches all possible uses
edge coverage
execute every branch
node coverage
execute every statement
FE
filtering effectiveness
pairwise testing
for each variable it is tested with every other value
software testing
formal process carried out by testing team, run the programs on computer. follows test procedures or test cases
test path
from initial node to final node
satisfaction
given a set TR of test requriments for criteria C, a set of tests T satisfies C on a graph iif for every test requirement in TR there is a test path in path T that meets the test requiremnt tr
Defect Removal Efficiency
gives a measure of the development team ability to remove defects prior to release. It is calculated as a ratio of defects resolved to total number of defects found. It is typically measured prior and at the moment of release.
def-use important!
id def and use appear on same node, then it is only a du if the def occurs after the use and the node is a loop
loops in graphs
if loop, complete path coverage not feasible, infinite # of paths
reach
if there is def clear path from i to j with var v then i reaches the use of j
Black box testing
ignores internal mechanisisms of system or component and focuses on outputs in response to give inputs. tests compliance with specified requirements
integration testing stratigies
incremental testing (bottom up, top down)or big ban testing (test al at once)
qualification process
is system or component suitible for operational use
formal design reviews
leader, review team of project memebers, customers, professionals outside project. YOu don't fix in design reviews. you sign off formally, approve document, verify and validate comments, decide action items. approve, partial approve, or denial of design review at end. DR report is summary of outcome
element of plan for software quality
list of quality goals, review activities, software tests, accepptance tests for external software, procedures and dates for versions
boundaries
mark the point or zone of transition from one EC to another
model's quantitiative result for DRE
measure total effectiveness at removing defects, and the total costs to remove defects
invalid EC
negative testing classes where you select only one value that is incorrect to make sure error is handled correctly
if statement
one node, two edges (for each condition), two nodes for block in each if statment, and then one node for end of if to return back to regular flow (diamond shaped)
du pairs
pairs of nodes from def to use
PD
passed defects
def-use
path from def to use
POD
phase originated defect
round trips
prime path that starts and ends at same node
factors that affect required intensity of sqa activities
project factors (technical complexity and difficulty, severity of failure outcomes, project magnitude), team factors (professional qualification of team members, availability, experience, familiarity (new team members)),
Objectives of having a development and quality plan
recruit team members, decide if waterfall or agile, resolve dev risks, milestones, sprint lengths, give mgmt data for project control
RD
removed defects
data flow coverage criteria
requires a graph to be annotated with references to variables
edge-pair coverage
requires edges or subpaths of length 2
loops in cfg
requires extra nodes added. while condition is met, arrow points back to initial node at end of block execution
test criteria
reules that define test requirements
best effort touring -
satisfy as many TR as possible. allow sidetrips to satisfy unsatisfiable test requirements
Classes of software dev risks
scheduling/timing risks, system functionality risks, subcontracting risks, requirement mgmt risk, resource usage and performance risk, personnel mgmt risk. must prioritize risks!
equivalence class boundaries
single numeric, alphabet value, group of alphanumerics, range of values, etc
Elements of plan for devs
specify deliverables, define project interfaces, methodolity and tools, milestones, map of dev process, cost estimate
nodes in cfg
statements or sequences of statements (basic blocks)
def
statements that assign values to variables
use
statements that use variables
2 testing harness/scaffold.
stubs and drivers
path(T)
test path is excuted by set of tests T
du-tours
test path tours subpath and the subpath is defclear with respect to v
Edge Coverage tR and Test path
test requirement - must contain every single edge at least once. must contain subpaths of length 1
node coverage
test requirement {123456} must contain every node once
node coverage
test set T satisfies node coerage iif every reachable node there is some path that visits the node
White box testing
testing that takes into account the internal mechanism of a system or component
return statements inside if block in cfg
the two choices will not converge and the return statment will be the finite state (double thick line)
combinatorial testing
to overcome short come of minimum test cases, can test every combination (while trying to minimize # test cases.)
TRC
total removal cost
complete path coverage
tr contains all paths in graph
complete round trip coverage
tr contains all round trop paths for each reachable node in g
all def coverage
tr contains at least one path from du-paths set. every def reaches a use
simple round trip coverage
tr contains at least one rount trip path for each reachable node that begins and ends in round trip path
edge coverage
tr contains each reachable path of length up to 1 in graph (single edge)/ special case of graph of a single node (a graph with one node will not have any edges)
edge pair coverage
tr contains each reachable path of length up to 2
specified graph coverage
tr contains set of test paths where s is a parameter
edges
transfers of control
infeasible test requirements
unreachable statement, subpath that requires contradiction (both > and < 0)
loops
whiles, for etc
line coverage
white box testing concept - percent of program code lines included in planned testing
Path coverage
whitebox testing concept. path coverage is measured by percentage of all possible program plaths included in planned testings