cmpe 187 final

A Functional Specification provides:

- A precise description of the major functions a system must fulfill the requirements - Explanation of the technological risks involved - External interfaces with other software modules - Data flow such as flowcharts, transaction sequence diagrams, and finite state machines describing the sequence of activities - Fault handling, memory utilization, and performance estimates - Any engineering limitation

Introduction

- Test project name - Revision history - Terminology and definitions - Name of the approvers and the date of approval - References - Summary of the rest of the test plan

Test Execution Strategy

- The process of the system test execution, defect detection, and fixing defects are intricately intertwined

Structure of a System Test Plan

1. Introduction 2. Feature description 3. Assumptions 4. Test Approach 5. Test suite structure 6. Test environment 7. Test execution strategy 8. Test effort estimation 9. Scheduling and milestones

Basic Idea in Software Process

A process comprises a set of activities that are executed to develop products Different processes are driven by different goals and availability of resources It is useful to follow a defined process because of the following benefits

Quality Control

A set of activities designed to evaluate the quality of developed or manufactured products

System Test Group

Ensure that the system requirements have been satisfied and that the system is acceptable

Second Definition

Failure intensity is a measure of the reliability of a software system operating in a given environment.

TMM: Level 1

Initial: no testing goals to be met tests are written after code Testing is not a critical to the software development process

TMM: Level 2

Phase Definition: The maturity goals are as follows Develop testing and debugging goals initiate a test planning process institutionalize basic testing techniques and methods

Test Prioritization in Test Cycle 1

Prioritize the test cases that allow the maximum number of test cases to execute.

Fault Count in a system is influenced by:

Size and complexity of code. Characteristics of the development process used. Education, experience, and training of development personnel Operational environment

Software Quality Assurance Groups

Software QA groups deal not only with the location of defects, but how to prevent defects

First Definition

Software reliability is defined as the probability of failure-free operation of a software system for a specified amount of time in a specific environment

Requirements Traceability

The ability to describe and follow the life of a requirement, in both forward and backward direction, i.e. from its origin through its development and specification, to its deployment and use, and through periods of refinement

Fault

The adjudged cause of a failure`

MTTR: Mean time to repair

The time between a failure occurrence and correction of the failure

MTTF: Meant time to failure

The time between two failures

Operational Profile

The ways that the user uses the system

Quality Mgmt Group

This group works on customizing software development processes and ensuring that processes are adhered to

Two Applications of a Traceability Matrix

To identify and track the functional coverage of a test. To identify which test cases must be exercised or updated when a system evolves.

Five Views of Software Quality

Transcendental User Manufacturing Product Value-based

Modeling Defect

Two key concepts involved in modeling defects are severity and priority

Requirements Identification

Two major challenges in identifying requirements: 1) Ensure the right requirements are captured which is essential for meeting the expectations of the user 2) Ensure that requirements are communicated unambiguously to the developers and testers so there are no surprises when system is delivered

Gantt Chart

Used to represent a project schedule

Functionally Related

Variables in the same assignment variable are functionally related. Variables in the same branch predicate are functionally related.

Equivalence Class Partitioning

When input domain is too large for testing, the input domain is partitioned into a finite number of subdomains - each subdomain is known as an equivalence class.

Two ways of counting reliability:

1) Counting failures in periodic intervals. 2) Failure intensity

Howden's Functional Testing Summary

1) Identify the input and output variables of the program and their data domains 2) Compute the expected outcomes for selected input values 3) Determine the input values that will cause the program to produce selected output values

Four Key Concepts of Functional testing:

1) Precisely identify the domain of each input and output variable. 2) Select values from the data domain of each variable having important qualities 3) Consider combinations of different input domains to design test cases 4) Consider input values such that the program under test produces special values from the domains of the output variables

Three major objectives of acceptance testing:

1. Confirm the system meets the accepted testing critieria 2. Identify and resolve discrepancies 3. Determine the readiness of the system for cut-over to live system

Point of Control and Observation

A PCO is a point of interaction between a system and its users

Failure

A failure is said to occur if the observable outcome of a program execution is different than the epected outcome

Acceptance Testing

A formal testing conducted to determine if the system satisfies its acceptance criteria. User Acceptance testing and Business Acceptance testing

What is reliability?

A system that rarely faults is considered to be reliable.

Test Oracle

A test oracle is a mechanism that verifies the correctness of program outputs.

Test Vector

A test vector in an instance of an input to a program, a.k.a test data.

The Capability Maturity Model

Allow an organization to evaluate it software developmental processes There are five levels of the CMM

Traceability Matrix

Allows one to find a two-way mapping requirements and test cases as follows: - From a requirement to a functional specification to specific tests which exercise the requirements - From each test case back to the requirement and functional specifications

Quality Criterion

An attribute of a quality factor that is related to software development.

Manufacturing view

Any deviation from the requirements detracts from the quality

Applications of Software Reliability

Comparison of software engineering technologies? Measuring the progress of system testing Controlling the system in operation Better insight into software development process

Beta Testing

Conducted by the potential buyers of the system

State-oriented Model

Control Portion and Data Portion.

Five C's of a quality engineer

Controlled, Comprehensive, Considerate, Critical, Competent

Assumptions

Describes the areas for which test cases will not be designed.

Test Approach

Describes: - Issues discovered by customers that were not caught during system testing in the past project are discussed and the preventive action that are being taken in this test project. - If there are any outstanding issues that need to be tested differently need to be discussed here - A test automation strategy for writing scripts is a topic of discussion - Identify test cases from the database that can be re-used in this test-plan - Give an outline of the tools, formats, and organizing scheme, such as traceability matrix, - First level of test cateogries.

Test Suite

Detail test groups and subgroups are outlined based on the test categories identified in the test approach section

Characterization of Test Cycles

GOALS: -System test team sets its own goals to be realized during test cycle -These goals are ideal in that they are very high standard -Goals are speicified in terms of the number of test cases to pass in a cycle ASSUMPTIONS: -How often the builds will be selected during a system test cycle? TEST EXECUTION: Prioritization of test execution chages between test cycles Test cases that exercise basic functionalities have higher priority

A software process comprises the following tasks:

Gathering requirements constructing functional specification designing the system writing code testing the system maintaining the system

Test cycle 2

Goal maximize number of passed test cases (99%)

Test cycle 3

Goal: all test cases should pass

Test cycle 1

Goal: maximize number of passed test cases (98%)

TMM: Level 3

Integration. Maturity goals in this level: Establish a software test group Establish a technical training Integrate testing nto the software lifecycle Control and monitor the testing process.

Revert and Extension Criteria

It may not be useful to continue a test cycle if it is found that a software is of poor quality

Unique Input/Output Sequence

Let X be an input sequence to a state s, and let Y be the output sequence. X/Y is a UIO sequence for s if no other state produces output sequence Y in response to input X

TMM: Level 4

MGMT and measurement: Establish an organization wide review program Establish a test management program Evaluate software quality

MTBF: Mean time between failures

MTTF + MTTR

Integration Test Group

Make sure that the unit-tested modules are interface correctly with each other.

Test Environment

Multiple test environments are constructed in practice. A schematic diagram of one or more test beds are presented in this section of the system test plan.

Different Types of Variables

Numeric, Arrays, Substructures, subroutine arguments

Product view

One can explore the causal relationships between internal properties and external qualities

TMM: Level 5

Optimization, Defect PRevention and Quality Control Application of process data for defect prevention Statistical quality control Test process optimization

Prioritization of Test Cases

Ordering the execution of test cases according to certain test objectives.

Pairwise Testing

Pairwise testing means that for each possible combination of input values for every pair of input variables is covered by at least one test case.

User View

Quality concerns the extent to which a product meets the users needs and expectations

Value based View

Quality is a measure of excellence, value is a measure of worth. How much is a customer willing to pay for a certain level of quality?

Transcendental View

Quality is something that can be recognized through experience, but not defined in some tractable form

Random Testing

Random testing gives the advantage of easily estimating software reliability from test outcomes.

Test Groups

Recommended to have at least two: System test group and integration test group

Life cycle of a defect

Represented by a state diagram with five states : NEW, ASSIGNED, OPEN, RESOLVED, CLOSED

Quality factors

Represents the behavioral characteristics of a system. (Correctness, reliability, efficiency

An intuitive approach to improving a test process is as follows:

Step 1) Determine an area for improvement Step 2) Evaluate the current status of the test process Step 3) IDentify the next desired state and th means Step 4) Implement the necessary changes to the process.

Selecting Test Cases for the Final Test Cycle

Step 1: The test suite is partitioned into red, yellow, green and white bins. RED: hodls test cases that failed previously; tests from scalability, stress, load and stability. YELLOW: Test cases that are useful to execute GREEN: Test cases that do not add any value to regression testing. WHITE: Test cases for which no concrete decision can be made in the first step Step 2: Move test cases from white bin to the other bins Step 3: Test cases from red and yellow bin are executed

Feature Description

Summarizes the high level description of the functionalities of the system.

Metrics for monitoring test execution

Test case Escapes: number of test cases escaped. high number means testing deficiencies. Planned versus actual execution. Execution status of test cases

Test process improvement

Test process needs to be improved for three reasons: Quality: a better test process should give more insights into the quality characteristics of the system being tested Lead Time: a better test process saves testing time Cost: A better test process is expected to be carried out with a lower cost

Test Design Factors

The following factors that must be taken into consideration during the design of system tests: 1) Coverage metrics 2) Effectiveness 3) Productivity 4) Validation 5) Maintenance 6) Users skills

Boundary Value Analysis

The idea of BVA is to select test data near the boundary of a data domain so that data both within and outside the boundary are tested

Test effort estimation

The number of test cases created by a person in a day; the number of test cases executed by a person in a day;

Structure of a System Test Plan

The purpose of a system test plan is as follows: - It provides guidance for the executive mgmt to support the test project - It establishes the foundation of a system testing part of the overall project - It provides assurance of test coverage by creating a requirement traceability matrix - It outlines an orderly schedule of events and test milestones that are tracked - It specifies the personnel, financial, equipment, and facility resources required to support the system testing part of the project

Defect causal analysis

focuses on understanding of cause-effect relatioship

Test prioritization of test cycle 3

similar to cycle 2, but subset of test are chosen for regression testing

Test prioritiation in test cycle 2

test cases that failed in first cycle are tested early in this cycle.