cmpe 187 final
A Functional Specification provides:
- A precise description of the major functions a system must fulfill the requirements - Explanation of the technological risks involved - External interfaces with other software modules - Data flow such as flowcharts, transaction sequence diagrams, and finite state machines describing the sequence of activities - Fault handling, memory utilization, and performance estimates - Any engineering limitation
Introduction
- Test project name - Revision history - Terminology and definitions - Name of the approvers and the date of approval - References - Summary of the rest of the test plan
Test Execution Strategy
- The process of the system test execution, defect detection, and fixing defects are intricately intertwined
Structure of a System Test Plan
1. Introduction 2. Feature description 3. Assumptions 4. Test Approach 5. Test suite structure 6. Test environment 7. Test execution strategy 8. Test effort estimation 9. Scheduling and milestones
Basic Idea in Software Process
A process comprises a set of activities that are executed to develop products Different processes are driven by different goals and availability of resources It is useful to follow a defined process because of the following benefits
Quality Control
A set of activities designed to evaluate the quality of developed or manufactured products
System Test Group
Ensure that the system requirements have been satisfied and that the system is acceptable
Second Definition
Failure intensity is a measure of the reliability of a software system operating in a given environment.
TMM: Level 1
Initial: no testing goals to be met tests are written after code Testing is not a critical to the software development process
TMM: Level 2
Phase Definition: The maturity goals are as follows Develop testing and debugging goals initiate a test planning process institutionalize basic testing techniques and methods
Test Prioritization in Test Cycle 1
Prioritize the test cases that allow the maximum number of test cases to execute.
Fault Count in a system is influenced by:
Size and complexity of code. Characteristics of the development process used. Education, experience, and training of development personnel Operational environment
Software Quality Assurance Groups
Software QA groups deal not only with the location of defects, but how to prevent defects
First Definition
Software reliability is defined as the probability of failure-free operation of a software system for a specified amount of time in a specific environment
Requirements Traceability
The ability to describe and follow the life of a requirement, in both forward and backward direction, i.e. from its origin through its development and specification, to its deployment and use, and through periods of refinement
Fault
The adjudged cause of a failure`
MTTR: Mean time to repair
The time between a failure occurrence and correction of the failure
MTTF: Meant time to failure
The time between two failures
Operational Profile
The ways that the user uses the system
Quality Mgmt Group
This group works on customizing software development processes and ensuring that processes are adhered to
Two Applications of a Traceability Matrix
To identify and track the functional coverage of a test. To identify which test cases must be exercised or updated when a system evolves.
Five Views of Software Quality
Transcendental User Manufacturing Product Value-based
Modeling Defect
Two key concepts involved in modeling defects are severity and priority
Requirements Identification
Two major challenges in identifying requirements: 1) Ensure the right requirements are captured which is essential for meeting the expectations of the user 2) Ensure that requirements are communicated unambiguously to the developers and testers so there are no surprises when system is delivered
Gantt Chart
Used to represent a project schedule
Functionally Related
Variables in the same assignment variable are functionally related. Variables in the same branch predicate are functionally related.
Equivalence Class Partitioning
When input domain is too large for testing, the input domain is partitioned into a finite number of subdomains - each subdomain is known as an equivalence class.
Two ways of counting reliability:
1) Counting failures in periodic intervals. 2) Failure intensity
Howden's Functional Testing Summary
1) Identify the input and output variables of the program and their data domains 2) Compute the expected outcomes for selected input values 3) Determine the input values that will cause the program to produce selected output values
Four Key Concepts of Functional testing:
1) Precisely identify the domain of each input and output variable. 2) Select values from the data domain of each variable having important qualities 3) Consider combinations of different input domains to design test cases 4) Consider input values such that the program under test produces special values from the domains of the output variables
Three major objectives of acceptance testing:
1. Confirm the system meets the accepted testing critieria 2. Identify and resolve discrepancies 3. Determine the readiness of the system for cut-over to live system
Point of Control and Observation
A PCO is a point of interaction between a system and its users
Failure
A failure is said to occur if the observable outcome of a program execution is different than the epected outcome
Acceptance Testing
A formal testing conducted to determine if the system satisfies its acceptance criteria. User Acceptance testing and Business Acceptance testing
What is reliability?
A system that rarely faults is considered to be reliable.
Test Oracle
A test oracle is a mechanism that verifies the correctness of program outputs.
Test Vector
A test vector in an instance of an input to a program, a.k.a test data.
The Capability Maturity Model
Allow an organization to evaluate it software developmental processes There are five levels of the CMM
Traceability Matrix
Allows one to find a two-way mapping requirements and test cases as follows: - From a requirement to a functional specification to specific tests which exercise the requirements - From each test case back to the requirement and functional specifications
Quality Criterion
An attribute of a quality factor that is related to software development.
Manufacturing view
Any deviation from the requirements detracts from the quality
Applications of Software Reliability
Comparison of software engineering technologies? Measuring the progress of system testing Controlling the system in operation Better insight into software development process
Beta Testing
Conducted by the potential buyers of the system
State-oriented Model
Control Portion and Data Portion.
Five C's of a quality engineer
Controlled, Comprehensive, Considerate, Critical, Competent
Assumptions
Describes the areas for which test cases will not be designed.
Test Approach
Describes: - Issues discovered by customers that were not caught during system testing in the past project are discussed and the preventive action that are being taken in this test project. - If there are any outstanding issues that need to be tested differently need to be discussed here - A test automation strategy for writing scripts is a topic of discussion - Identify test cases from the database that can be re-used in this test-plan - Give an outline of the tools, formats, and organizing scheme, such as traceability matrix, - First level of test cateogries.
Test Suite
Detail test groups and subgroups are outlined based on the test categories identified in the test approach section
Characterization of Test Cycles
GOALS: -System test team sets its own goals to be realized during test cycle -These goals are ideal in that they are very high standard -Goals are speicified in terms of the number of test cases to pass in a cycle ASSUMPTIONS: -How often the builds will be selected during a system test cycle? TEST EXECUTION: Prioritization of test execution chages between test cycles Test cases that exercise basic functionalities have higher priority
A software process comprises the following tasks:
Gathering requirements constructing functional specification designing the system writing code testing the system maintaining the system
Test cycle 2
Goal maximize number of passed test cases (99%)
Test cycle 3
Goal: all test cases should pass
Test cycle 1
Goal: maximize number of passed test cases (98%)
TMM: Level 3
Integration. Maturity goals in this level: Establish a software test group Establish a technical training Integrate testing nto the software lifecycle Control and monitor the testing process.
Revert and Extension Criteria
It may not be useful to continue a test cycle if it is found that a software is of poor quality
Unique Input/Output Sequence
Let X be an input sequence to a state s, and let Y be the output sequence. X/Y is a UIO sequence for s if no other state produces output sequence Y in response to input X
TMM: Level 4
MGMT and measurement: Establish an organization wide review program Establish a test management program Evaluate software quality
MTBF: Mean time between failures
MTTF + MTTR
Integration Test Group
Make sure that the unit-tested modules are interface correctly with each other.
Test Environment
Multiple test environments are constructed in practice. A schematic diagram of one or more test beds are presented in this section of the system test plan.
Different Types of Variables
Numeric, Arrays, Substructures, subroutine arguments
Product view
One can explore the causal relationships between internal properties and external qualities
TMM: Level 5
Optimization, Defect PRevention and Quality Control Application of process data for defect prevention Statistical quality control Test process optimization
Prioritization of Test Cases
Ordering the execution of test cases according to certain test objectives.
Pairwise Testing
Pairwise testing means that for each possible combination of input values for every pair of input variables is covered by at least one test case.
User View
Quality concerns the extent to which a product meets the users needs and expectations
Value based View
Quality is a measure of excellence, value is a measure of worth. How much is a customer willing to pay for a certain level of quality?
Transcendental View
Quality is something that can be recognized through experience, but not defined in some tractable form
Random Testing
Random testing gives the advantage of easily estimating software reliability from test outcomes.
Test Groups
Recommended to have at least two: System test group and integration test group
Life cycle of a defect
Represented by a state diagram with five states : NEW, ASSIGNED, OPEN, RESOLVED, CLOSED
Quality factors
Represents the behavioral characteristics of a system. (Correctness, reliability, efficiency
An intuitive approach to improving a test process is as follows:
Step 1) Determine an area for improvement Step 2) Evaluate the current status of the test process Step 3) IDentify the next desired state and th means Step 4) Implement the necessary changes to the process.
Selecting Test Cases for the Final Test Cycle
Step 1: The test suite is partitioned into red, yellow, green and white bins. RED: hodls test cases that failed previously; tests from scalability, stress, load and stability. YELLOW: Test cases that are useful to execute GREEN: Test cases that do not add any value to regression testing. WHITE: Test cases for which no concrete decision can be made in the first step Step 2: Move test cases from white bin to the other bins Step 3: Test cases from red and yellow bin are executed
Feature Description
Summarizes the high level description of the functionalities of the system.
Metrics for monitoring test execution
Test case Escapes: number of test cases escaped. high number means testing deficiencies. Planned versus actual execution. Execution status of test cases
Test process improvement
Test process needs to be improved for three reasons: Quality: a better test process should give more insights into the quality characteristics of the system being tested Lead Time: a better test process saves testing time Cost: A better test process is expected to be carried out with a lower cost
Test Design Factors
The following factors that must be taken into consideration during the design of system tests: 1) Coverage metrics 2) Effectiveness 3) Productivity 4) Validation 5) Maintenance 6) Users skills
Boundary Value Analysis
The idea of BVA is to select test data near the boundary of a data domain so that data both within and outside the boundary are tested
Test effort estimation
The number of test cases created by a person in a day; the number of test cases executed by a person in a day;
Structure of a System Test Plan
The purpose of a system test plan is as follows: - It provides guidance for the executive mgmt to support the test project - It establishes the foundation of a system testing part of the overall project - It provides assurance of test coverage by creating a requirement traceability matrix - It outlines an orderly schedule of events and test milestones that are tracked - It specifies the personnel, financial, equipment, and facility resources required to support the system testing part of the project
Defect causal analysis
focuses on understanding of cause-effect relatioship
Test prioritization of test cycle 3
similar to cycle 2, but subset of test are chosen for regression testing
Test prioritiation in test cycle 2
test cases that failed in first cycle are tested early in this cycle.