Software Testing & QA - Midterm

3 (Requirements)

"The lab machine shall be accessible by only authorized personnel" is an example of 1. Environment assertions 2. Specifications 3. Requirements

Black Box Testing

(BBT vs WBT) Defect focus is failures

White Box Testing

(BBT vs WBT) Defect focus is faults

Black Box Testing

(BBT vs WBT) Testers are non devs (IV&V)

White Box Testing

(BBT vs WBT) Testers are the developers of the software

Black Box Testing

(BBT vs WBT) Testing focuses on large software (as a whole)

White Box Testing

(BBT vs WBT) Testing focuses on small objects (looking inside)

White Box Testing

(BBT vs WBT) Testing timeline is earlier (unit testing).

Black Box Testing

(BBT vs WBT) Testing timeline is later (acceptance stage).

Quality Assurance

(IMPORTANT) To ensure that few, if any, defects remain in the software when it is delivered to its customers or released to the market

1 (Environment assertions)

- Auth. person has username - Auth. person has password - Passwords are never shared with non-authorized personnel 1. Environment assertions 2. Specifications 3. Requirements

Operational Profile

- Definition: a list of disjoint set of operations and their associated probabilities of occurrence - A quantitative way of characterizing the way a software system is or will be used - Operations: multiple possible test cases or multiple runs • Each operation corresponds to an individual sub-domain in domain partitions, thus representing a whole equivalence class.

OP: Operational Profile

- Quantitative characterization of the way a system will be used - Generate/execute test cases for UBST - Realistic reliability assessment - development decisions/priorities

Coverage-Based Testing

- Systematic testing based on formal models and techniques - Testing models based on internal details or external expectations - Coverage measures defined for models - Testing measured by coverage goals - Example: Canvas / unit-testing: post announcement & set receiver role(s); set up submission deadline; ...

Partition testing

-Input data and output results often fall into different classes where all members of a class are related. -Each of these classes is an equivalence partition or domain where the program behaves in an equivalent way for each class member. -Test cases should be chosen from each partition.

Error

A mistake made by a programmer or software engineer which caused the fault, which in turn may cause a failure

2 (Specifications)

Access to the lab machine shall be granted only after the user types an authorized "username, password" pair 1. Environment assertions 2. Specifications 3. Requirements

card insertion & rejection password management envelope and printing abnormal termination installation and re-installation backup and restore commit and rollback locking logging and recovery migration stress

An example high-level functional checklist for an ATM

All stages of testing (Particularly unit and component testing - Later phases at high abstraction levels)

Applicability of Coverage-Based Testing

- Final stages of testing - Particularly system/acceptance testing - Use with software reliability engineering

Applicability of Usage-Based (Statistical) Testing

Domain properties and requirements (real world)

Application Domain

- Functional/external/black-box - Structural/internal/white-box

Broadly, what types of testing do we use?

no

Can a system be error free forever?

Fault

Condition / internal characteristic that may cause a failure in the system

Does a certain number of things correctly

Coverage based testing

White

Coverage-Based Testing is a (Black/White) box testing type?

small objects

Coverage-Based Testing is more suited to (large software/small objects)?

Testing (operational) Definition

Defined as: the execution of software and the observation of its behavior or outcome demonstrated using controlled experiments

- Pure membership based partitions (+ or - numbers) - Properties/relations used in definitions (x <= 100) - Combinations (+ or - numbers <= 100)

Different types of partitions

- Lack of structure - Likely to miss - Likely to repeat oneself - In general, the whole process is hard to repeat

Drawbacks of ad hoc testing?

equivalence class

Each G1...Gn in a partition is called an

Systematic Testing: Checklists

Examples - Functional (black-box) - System elements (white-box) - Structures (implementation/white-box) - Properties (black-box or white-box)

Add item to cart View cart Remove item from cart Add multiple of one item to cart Enter address Enter payment information Payment processing Confirmation email with unique number

Exercise: Create a functional checklist for an online shopping cart

Ability to support file types Upload file to canvas Ability to re-submit an assignment Submitting assignment not allowed after due date

Exercise: Create a functional checklist for submitting an assignment on canvas

Correctness

Few problems with limited damage to customers.

Black box

Function/feature (external) checklists are ___ in nature.

John D. Musa

Giant in SRE (SW Reliability Eng.) He is the reason we have the operational profile concept.

By using a systematic, disciplined and quantifiable approach

How do we achieve quality software (product)?

white box (but vary in levels of abstraction)

Implementation checklists are ___ in nature.

requirements

In RE, testing is driven by...?

- Reflexive - holds on every member - Symmetric - holds if order is change - Transitive - holds in a relation chain

In an equivalence class, the specific relation that is used to define the subsets is:

holes in the data (testing for postitive intergers 1 - 100 but missed the number 77 somehow)

In partitioning, collectively exhaustive equivalence classes eliminates...?

duplicates in the data set

In partitioning, mutually exclusive equivalence classes eliminates...?

defining specific input variables and associated values to exercise certain parts of the program in the white-box view or to perform certain functions in the black-box view e.g., function add(int a, int b) considering valid/invalid input values of a and b • How many cases are in an exhaustive test?

In regards to partition coverage, what does it mean to "sensitize" test cases?

Failure

Inability of the system to perform a function according to its specification due to some fault. (- deviation from expected behavior - something goes wrong at execution)

Testing with checklists

List of items that must be tested -> Each item is "checked off" -> When list is complete, testing is done

Black Box Testing (Functional Testing)

Look at program spec to create test data covering I/O and program functions. Independently of code.

Computers and programs.

Machine domain

checklist

One way to add structure to testing is to build a ___

equivalent Sampling uniform

Partitions-Based Testing Basic ideas - Members in equivalence class are treated "___" ->Defining meaningful partitions - ___ from partitioned subsets for different types of partitions - Coverage of partitions is ___

- Sets are mutually exclusive - Sets are collectively exhaustive

Partitions: Formal Definitions A set S contains a list of unique elements A partition of S creates subsets G1, G2, ... Gn such that...?

mutually exclusive 2.

Partitions: Formal Definitions Given a set S of elements (say, integers), a partition of said set S into G1 and G2 should have no elements from G1 overlap into G2. Name this property and choose the below correct partition. Given S = 1, 2, 3, 4, 5, 6, 7, 8 1. G1: 1, 2, 3, 4, 5, | G2: 2, 4, 6, 7, 8 2. G1: 1, 2, 3, 4, | G2: 5, 6, 7, 8,

collectively exhaustive 1. (2. is missing the 5)

Partitions: Formal Definitions Given a set S of elements (say, integers), a partition of said set S into G1 and G2 should include all elements of S. Name this property and choose the below correct partition. Given S = 1, 2, 3, 4, 5, 6, 7, 8 1. G1: 8, 3, 5, 2 | G2: 1, 4, 6, 7 2. G1: 1, 2, 3, 4, | G2: 6, 7, 8,

Reliability

Probability of failure-free operation for a specific period or a given set of input under a specific environment

ok

R: requirements (optative/desired) ε: environment assertions (indicative/given) S: specifications (optative/desired)

ok

Reliability is accomplished through availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning (just type ok when done)

Mean-time-to-Failure (MTTF)

Reliability: Key Measures - System Quality: "How often does the thing stop working?"

Mean-time-to-Repair (MTTR)

Reliability: Key Measures - System Quality: "Once it stops working, how long does it take to fix it?"

- Check/validate the requirements for: • Correctness • Completeness • Ambiguity • Logical Consistency - Designing a set of test cases • For those requirements • From a black-box perspective

Requirements-Based Testing Addresses 2 major issues:

Michael Jackson

Software Requirements and Design: A Tribute to...?

Bridge between the application domain and the machine domain.

Specification

Those who have a stake in the change being considered &who stand to gain or lose from the change

Stakeholders

if you write a program that perfectly follows specifications (s), and the accepted truths (e) remains true, only then will the program deliver the requirements as desired

THE meaning of requirements

coverage goals

Termination criteria for Coverage-Based Testing

reliability goals (time to fail, and time to fix)

Termination criteria for Usage-Based (Statistical) Testing

White Box Testing (Structural Testing)

Testing based on an analysis of the internal structure of the component or system.

Pass, fail, describe requirement improvement with justification

Testing: RBT Approach How do you Verify test results?

two types: resource based and quality based

Testing: RBT Approach How do you define test completion criteria?

a) Structure/Formalize requirements b) Build/Define test cases

Testing: RBT Approach How do you design test cases (define logical test cases)?

Think about "Correctness and Completeness" • But How? • Think about "Environment Assertions"

Testing: RBT Approach How do you perform a domain expert review?

a) Validate against business objectives (Usefulness, Novelty) b) Ambiguity analysis / Domain Expert review

Testing: RBT Approach How do you validate a requirement?

Customer view of quality

UBST ensures reliability • Reliability:

resource depletion quality achieved

Under what two circumstances should you stop testing?

Don't waste time testing everything, focus on what will actually be used

Usage based testing

randomly sample

Usage-Based (Statistical) Testing Simulated in testing environment there are too many possible environments (think of all the browsers and OS versions out there) so we ___ for testing environment.

large software

Usage-Based (Statistical) Testing is more suited to (large software/small objects)?

reliability

Usage-Based Statistical Testing ensures ___

Black (we're only concerned with the external behavior of a system)

Usage-Based Statistical Testing is a (Black/White) box testing type?

Waterfall Model V-Model Spiral Model eXtreme Programming

What are process (lifecycle) models?

Does what it's supposed to, how it's supposed to, reliably. (- does what it is supposed to do - does the things in a desired way - show/demonstrate/prove the above two points)

What is "quality"?

A relation from a domain X to a codomain Y such that for every element x in X, there is a unique y=f(x) in Y.

What is a "function" as in mathematics?

a mistake written down in code and/or document ( - e.g., if(current_enroll = max_enrol) {//cannot enroll any more} - SHOULD BE if(current_enroll == max_enrol) {//cannot enroll any more} )

What is a fault?

conceptual mistakes made by a human (• e.g., this is a Linux App. So it is going to be more secure anyway.)

What is an error?

Coding standard: - Naming conventions: to improve software maintainability - e.g., standard items (in concurrency control) • ACID (atomicity, consistency, isolation, durability) • Locking (e.g., read-lock, write-lock, two-phase) • Serialization (timestamp ordering, commit ordering, etc.)

What is an example of an implementation checklist?

What should happen. Expected output given a certain input.

What is an oracle?

Ad hoc testing

What type of testing? - "run-and-observe" - Implicit checklists may be involved

A difference between things as desired and things as perceived.

What's a "problem"?

Grading scheme for a course. [0, 60), [60, 70), [70, 80), [80, 90), [90, 100]

What's a partition? Can you give an example?

Usage-based

Which testing method would be better suited to testing Graphical Interface type (websites and such)? Coverage-based or Usage-based?

Coverage-based

Which testing method would be better suited to testing an ATM? Coverage-based or Usage-based?

Functional (black-box) testing

Which type of testing tests external functions?

Structural (white-box) testing

Which type of testing tests internal implementations?

Deliver quality software (To solve customer's problem via software-intensive systems - To produce quality software within budget and schedule - To show/demonstrate/prove the software is indeed high quality)

Why do we engineer software?

defect detection & reduction (- Failure observation -> fault removal - Fault detection -> fault fixing - Error identification -> error prevention)

Why do we test software?

Developer Issues

conceptual mistakes, unfamiliarity with domain, inexperience with methods

S

is a description of the behaviors that the program must have in order to meet the requirements • Can only be written in terms of shared phenomena!

Systematic

means the (testing) process is explicitly defined

Requirements

stakeholders' needs and desires

ε

things in the environment that are true whether or not we ever build the proposed system

R

things in the environment that we wish to be made true by delivering the proposed system • Many of which will involve phenomena to which the machine has no access