Software Testing & QA - Midterm
3 (Requirements)
"The lab machine shall be accessible by only authorized personnel" is an example of 1. Environment assertions 2. Specifications 3. Requirements
Black Box Testing
(BBT vs WBT) Defect focus is failures
White Box Testing
(BBT vs WBT) Defect focus is faults
Black Box Testing
(BBT vs WBT) Testers are non devs (IV&V)
White Box Testing
(BBT vs WBT) Testers are the developers of the software
Black Box Testing
(BBT vs WBT) Testing focuses on large software (as a whole)
White Box Testing
(BBT vs WBT) Testing focuses on small objects (looking inside)
White Box Testing
(BBT vs WBT) Testing timeline is earlier (unit testing).
Black Box Testing
(BBT vs WBT) Testing timeline is later (acceptance stage).
Quality Assurance
(IMPORTANT) To ensure that few, if any, defects remain in the software when it is delivered to its customers or released to the market
1 (Environment assertions)
- Auth. person has username - Auth. person has password - Passwords are never shared with non-authorized personnel 1. Environment assertions 2. Specifications 3. Requirements
Operational Profile
- Definition: a list of disjoint set of operations and their associated probabilities of occurrence - A quantitative way of characterizing the way a software system is or will be used - Operations: multiple possible test cases or multiple runs • Each operation corresponds to an individual sub-domain in domain partitions, thus representing a whole equivalence class.
OP: Operational Profile
- Quantitative characterization of the way a system will be used - Generate/execute test cases for UBST - Realistic reliability assessment - development decisions/priorities
Coverage-Based Testing
- Systematic testing based on formal models and techniques - Testing models based on internal details or external expectations - Coverage measures defined for models - Testing measured by coverage goals - Example: Canvas / unit-testing: post announcement & set receiver role(s); set up submission deadline; ...
Partition testing
-Input data and output results often fall into different classes where all members of a class are related. -Each of these classes is an equivalence partition or domain where the program behaves in an equivalent way for each class member. -Test cases should be chosen from each partition.
Error
A mistake made by a programmer or software engineer which caused the fault, which in turn may cause a failure
2 (Specifications)
Access to the lab machine shall be granted only after the user types an authorized "username, password" pair 1. Environment assertions 2. Specifications 3. Requirements
card insertion & rejection password management envelope and printing abnormal termination installation and re-installation backup and restore commit and rollback locking logging and recovery migration stress
An example high-level functional checklist for an ATM
All stages of testing (Particularly unit and component testing - Later phases at high abstraction levels)
Applicability of Coverage-Based Testing
- Final stages of testing - Particularly system/acceptance testing - Use with software reliability engineering
Applicability of Usage-Based (Statistical) Testing
Domain properties and requirements (real world)
Application Domain
- Functional/external/black-box - Structural/internal/white-box
Broadly, what types of testing do we use?
no
Can a system be error free forever?
Fault
Condition / internal characteristic that may cause a failure in the system
Does a certain number of things correctly
Coverage based testing
White
Coverage-Based Testing is a (Black/White) box testing type?
small objects
Coverage-Based Testing is more suited to (large software/small objects)?
Testing (operational) Definition
Defined as: the execution of software and the observation of its behavior or outcome demonstrated using controlled experiments
- Pure membership based partitions (+ or - numbers) - Properties/relations used in definitions (x <= 100) - Combinations (+ or - numbers <= 100)
Different types of partitions
- Lack of structure - Likely to miss - Likely to repeat oneself - In general, the whole process is hard to repeat
Drawbacks of ad hoc testing?
equivalence class
Each G1...Gn in a partition is called an
Systematic Testing: Checklists
Examples - Functional (black-box) - System elements (white-box) - Structures (implementation/white-box) - Properties (black-box or white-box)
Add item to cart View cart Remove item from cart Add multiple of one item to cart Enter address Enter payment information Payment processing Confirmation email with unique number
Exercise: Create a functional checklist for an online shopping cart
Ability to support file types Upload file to canvas Ability to re-submit an assignment Submitting assignment not allowed after due date
Exercise: Create a functional checklist for submitting an assignment on canvas
Correctness
Few problems with limited damage to customers.
Black box
Function/feature (external) checklists are ___ in nature.
John D. Musa
Giant in SRE (SW Reliability Eng.) He is the reason we have the operational profile concept.
By using a systematic, disciplined and quantifiable approach
How do we achieve quality software (product)?
white box (but vary in levels of abstraction)
Implementation checklists are ___ in nature.
requirements
In RE, testing is driven by...?
- Reflexive - holds on every member - Symmetric - holds if order is change - Transitive - holds in a relation chain
In an equivalence class, the specific relation that is used to define the subsets is:
holes in the data (testing for postitive intergers 1 - 100 but missed the number 77 somehow)
In partitioning, collectively exhaustive equivalence classes eliminates...?
duplicates in the data set
In partitioning, mutually exclusive equivalence classes eliminates...?
defining specific input variables and associated values to exercise certain parts of the program in the white-box view or to perform certain functions in the black-box view e.g., function add(int a, int b) considering valid/invalid input values of a and b • How many cases are in an exhaustive test?
In regards to partition coverage, what does it mean to "sensitize" test cases?
Failure
Inability of the system to perform a function according to its specification due to some fault. (- deviation from expected behavior - something goes wrong at execution)
Testing with checklists
List of items that must be tested -> Each item is "checked off" -> When list is complete, testing is done
Black Box Testing (Functional Testing)
Look at program spec to create test data covering I/O and program functions. Independently of code.
Computers and programs.
Machine domain
checklist
One way to add structure to testing is to build a ___
equivalent Sampling uniform
Partitions-Based Testing Basic ideas - Members in equivalence class are treated "___" ->Defining meaningful partitions - ___ from partitioned subsets for different types of partitions - Coverage of partitions is ___
- Sets are mutually exclusive - Sets are collectively exhaustive
Partitions: Formal Definitions A set S contains a list of unique elements A partition of S creates subsets G1, G2, ... Gn such that...?
mutually exclusive 2.
Partitions: Formal Definitions Given a set S of elements (say, integers), a partition of said set S into G1 and G2 should have no elements from G1 overlap into G2. Name this property and choose the below correct partition. Given S = 1, 2, 3, 4, 5, 6, 7, 8 1. G1: 1, 2, 3, 4, 5, | G2: 2, 4, 6, 7, 8 2. G1: 1, 2, 3, 4, | G2: 5, 6, 7, 8,
collectively exhaustive 1. (2. is missing the 5)
Partitions: Formal Definitions Given a set S of elements (say, integers), a partition of said set S into G1 and G2 should include all elements of S. Name this property and choose the below correct partition. Given S = 1, 2, 3, 4, 5, 6, 7, 8 1. G1: 8, 3, 5, 2 | G2: 1, 4, 6, 7 2. G1: 1, 2, 3, 4, | G2: 6, 7, 8,
Reliability
Probability of failure-free operation for a specific period or a given set of input under a specific environment
ok
R: requirements (optative/desired) ε: environment assertions (indicative/given) S: specifications (optative/desired)
ok
Reliability is accomplished through availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning (just type ok when done)
Mean-time-to-Failure (MTTF)
Reliability: Key Measures - System Quality: "How often does the thing stop working?"
Mean-time-to-Repair (MTTR)
Reliability: Key Measures - System Quality: "Once it stops working, how long does it take to fix it?"
- Check/validate the requirements for: • Correctness • Completeness • Ambiguity • Logical Consistency - Designing a set of test cases • For those requirements • From a black-box perspective
Requirements-Based Testing Addresses 2 major issues:
Michael Jackson
Software Requirements and Design: A Tribute to...?
Bridge between the application domain and the machine domain.
Specification
Those who have a stake in the change being considered &who stand to gain or lose from the change
Stakeholders
if you write a program that perfectly follows specifications (s), and the accepted truths (e) remains true, only then will the program deliver the requirements as desired
THE meaning of requirements
coverage goals
Termination criteria for Coverage-Based Testing
reliability goals (time to fail, and time to fix)
Termination criteria for Usage-Based (Statistical) Testing
White Box Testing (Structural Testing)
Testing based on an analysis of the internal structure of the component or system.
Pass, fail, describe requirement improvement with justification
Testing: RBT Approach How do you Verify test results?
two types: resource based and quality based
Testing: RBT Approach How do you define test completion criteria?
a) Structure/Formalize requirements b) Build/Define test cases
Testing: RBT Approach How do you design test cases (define logical test cases)?
Think about "Correctness and Completeness" • But How? • Think about "Environment Assertions"
Testing: RBT Approach How do you perform a domain expert review?
a) Validate against business objectives (Usefulness, Novelty) b) Ambiguity analysis / Domain Expert review
Testing: RBT Approach How do you validate a requirement?
Customer view of quality
UBST ensures reliability • Reliability:
resource depletion quality achieved
Under what two circumstances should you stop testing?
Don't waste time testing everything, focus on what will actually be used
Usage based testing
randomly sample
Usage-Based (Statistical) Testing Simulated in testing environment there are too many possible environments (think of all the browsers and OS versions out there) so we ___ for testing environment.
large software
Usage-Based (Statistical) Testing is more suited to (large software/small objects)?
reliability
Usage-Based Statistical Testing ensures ___
Black (we're only concerned with the external behavior of a system)
Usage-Based Statistical Testing is a (Black/White) box testing type?
Waterfall Model V-Model Spiral Model eXtreme Programming
What are process (lifecycle) models?
Does what it's supposed to, how it's supposed to, reliably. (- does what it is supposed to do - does the things in a desired way - show/demonstrate/prove the above two points)
What is "quality"?
A relation from a domain X to a codomain Y such that for every element x in X, there is a unique y=f(x) in Y.
What is a "function" as in mathematics?
a mistake written down in code and/or document ( - e.g., if(current_enroll = max_enrol) {//cannot enroll any more} - SHOULD BE if(current_enroll == max_enrol) {//cannot enroll any more} )
What is a fault?
conceptual mistakes made by a human (• e.g., this is a Linux App. So it is going to be more secure anyway.)
What is an error?
Coding standard: - Naming conventions: to improve software maintainability - e.g., standard items (in concurrency control) • ACID (atomicity, consistency, isolation, durability) • Locking (e.g., read-lock, write-lock, two-phase) • Serialization (timestamp ordering, commit ordering, etc.)
What is an example of an implementation checklist?
What should happen. Expected output given a certain input.
What is an oracle?
Ad hoc testing
What type of testing? - "run-and-observe" - Implicit checklists may be involved
A difference between things as desired and things as perceived.
What's a "problem"?
Grading scheme for a course. [0, 60), [60, 70), [70, 80), [80, 90), [90, 100]
What's a partition? Can you give an example?
Usage-based
Which testing method would be better suited to testing Graphical Interface type (websites and such)? Coverage-based or Usage-based?
Coverage-based
Which testing method would be better suited to testing an ATM? Coverage-based or Usage-based?
Functional (black-box) testing
Which type of testing tests external functions?
Structural (white-box) testing
Which type of testing tests internal implementations?
Deliver quality software (To solve customer's problem via software-intensive systems - To produce quality software within budget and schedule - To show/demonstrate/prove the software is indeed high quality)
Why do we engineer software?
defect detection & reduction (- Failure observation -> fault removal - Fault detection -> fault fixing - Error identification -> error prevention)
Why do we test software?
Developer Issues
conceptual mistakes, unfamiliarity with domain, inexperience with methods
S
is a description of the behaviors that the program must have in order to meet the requirements • Can only be written in terms of shared phenomena!
Systematic
means the (testing) process is explicitly defined
Requirements
stakeholders' needs and desires
ε
things in the environment that are true whether or not we ever build the proposed system
R
things in the environment that we wish to be made true by delivering the proposed system • Many of which will involve phenomena to which the machine has no access
