Lecture 1: Fundamentals of Software Testing

State-less System

- A compiler is a stateless system. - Test cases are very simple - Outcome depends solely on the current input.

Failure

- A failure is said to occur whenever the external behavior of a system does not conform to that prescribed in the system specification.

state-oriented

- ATM is a state-oriented system - Test cases are not that simple. A test case may consist of a sequences of <input, expected outcome> - The outcome depends both on the current state of the system and the current input. - ATM example: < check balance, $500.00 >, < withdraw, "amount?" >, < $200.00, "$200.00" >, < check balance, $300.00 >

Dynamic Analysis

- Actual program execution to expose possible program failure. - One observe some representative program behavior, and reach conclusion about the quality of the system.

Error

- An error is a state of the system. - An error state could lead to a failure in the absence of any corrective action by the system.

Lean principle

- An innovative principle to compress the time period from customer order to banking payment. - The objective was to minimize the consumption of resources that added no value to a product.

Check

- Assess the measurements and report the results to decision makers

Act

- Decide on changes needed to improve the process.

Plan

- Establish the objective and process to deliver the results

Verification

- Evaluation of software system that help in determining whether the product of a given development phase satisfies the requirements established before the start of that phase. • Building the product correctly.

Quality Models

- ISO 9126 - CMM - TPI - TMM

Do

- Implement the plan and measure its performance.

The Objectives of Testing

- It does work - It does not work - Reduce the risk of failures - Reduce the cost of testing

Static Analysis

- It examines the code and reasons over all behaviors that might arise during run time. Ex: Code review, inspection, and algorithm analysis

The Notion of Software Reliability

- It is defined as the probability of failure-free operation of a software system for a specified time in a specified environment. - It can be estimated via random testing - Test data must be drawn from the input distribution to closely resemble the future usage of the system. - Future usage pattern of a system is described in a form called operational profile

Defect

- It is synonymous of fault. - It a.k.a. bug.

SQC methods

- Pareto analysis - Trend Chart - Flow chart - Histogram - Scatter diagram - Control chart - Cause and effect diagram

TQC (Total Quality Control)

- Quality comes first, not the short-term profits - The customer comes first, not the producer. - Decisions are based on facts and data. - Management is participatory and respectful of all employees. - Management is driven by cross-functional committees.

Role of Testing

- Software quality assessment divides into two categories: • Static Analysis • Dynamic Analysis - Static and Dynamic Analysis are complementary in nature. - Focus is to combines the strengths of both approaches.

Quality Revolution

- Stated in Japan by Deming, Juran, and Ishikawa during 1940s. - Statistical quality control (SQC). - Lean principle developed by Taiichi Ohno of Toyota.

What is a Test Case?

- Test Case is a simple pair of <input, expected outcome> - State-less systems - State-oriented

Views of software Quality

- Transcendental - User - Manufacturing: - Product - Value-based

Acceptance Testing

-Customer's expectations from the system. - Two types of acceptance testing: •User Acceptance Testing (UAT) •Business Acceptance Testing (BAT) -UAT: System satisfies the contractual acceptance criteria. -BAT: System will eventually pass the user acceptance test - performed by supplier's development organization.

Validation

-Evaluation of software systems that help in determining whether the product meets its intended use. • Building the correct product.

TQC methodology

Materials \ Machines \ ----------------------------------> Quality (Effect) Methods / Measurements/

PDCA cycle

Plan, Do, Check, Act

Regression Testing

Unit Testing --> Integration Testing --> System Testing --> Acceptance Testing •New test cases are not designed. •Tests are selected, prioritized and executed. •To ensure that nothing is broken in the new version of the software.

system testing

-Includes wide spectrum of testing such as functionality, and load.

Unit Testing

-Individual program units, such as procedure, methods in isolation.

Integration Testing

-Modules are assembled to construct larger subsystem and tested.

Complete Testing meaning

-The domain of possible inputs of a program is too large •Valid inputs •Invalid inputs -The design issues may be too complex to completely test. -It may not be possible to create all possible execution environments of the system.

Test Team Organization and Management

1- Executive Management 2- 2.1-Software Group 2.2-System Test Group 2.3-Hardware Group 3- under 2.1: Integration Test Group -- Software Developers 4- Development Test Group -- Performance Test Group --Scalability Test Group -- Automation Test Group --Sustaining Test Group

Fault

A fault is the adjudged cause of an error.

Expected Outcome

•An outcome of program execution may include: -Value produced by the program. - State Change. -A sequence of values which must be interpreted together for the outcome to be valid. •A test oracle is a mechanism that verifies the correctness of program outputs. -Generate expected results for the test inputs. -Compare the expected results with the actual results of execution of the IUT.

Black-box Testing

•Black-box testing a.k.a. functional testing. •Examines the program that is accessible from outside. •Applies the input to a program and observe the externally visible outcome. •It is applied to both an entire program as well as to individual program units. •It is performed at the external interface level of a system. •It is conducted by a separate software quality assurance group.

Central Issue in Testing

•Discovering all faults, is highly desirable, it is a near-impossible task, and it may not be attempted. •The next best thing is to select a subset of the input domain to test a program.

Test Design

•Each test case is designed as a combination of modular test components called test steps. •Test steps are combined together to create more complex tests.

Testing Activities

•Identify the objective to be tested. •Select inputs. •Compute the expected outcome. •Set up the execution environment of the program. •Execute the program. •Analyze the test results.

Test Tools and Automation

•Increased productivity of the testers. •Better coverage of regression testing. •Reduced durations of the testing phases. •Reduced cost of software maintenance. •Increased effectiveness of test cases. •The test cases to be automated are well defined. •Test tools and an infrastructure are in place. •The test automation professionals have prior successful experience in automation. Adequate budget have been allocation for the procurement of software tools

Monitoring and Measuring Test Execution

•Metrics for monitoring test execution. •Metrics for monitoring defects. •Test case effectiveness metrics. -Measure the "defect revealing ability" of the test suite. -Use the metric to improve the test design process. •Test-effort effectiveness metrics. -Number of defects found by the customers that were not found by the test engineers.

Source of Information for Test Selection

•Requirement and Functional Specifications. •Source Code. •Input and Output Domain. •Operational Profile. •Fault Model -Error Guessing -Fault Seeding -Mutation Analysis

Test Planning

•The purpose is to get ready and organized for test execution. •A test plan provides a: -Framework •A set of ideas, facts or circumstances within which the tests will be conducted. -Scope • The domain or extent of the test activities. -Details of resource needed -Effort required -Schedule of activities -Budget

Testing Level

•Unit Testing •Integration Testing •System Testing •Acceptance Testing

White Box Testing

•White-box testing a.k.a. structural testing. •Examines source code with focus on: -Control flow -Data flow •Control flow refers to flow of control from one instruction to another. •Data flow refers to propagation of values from one variable or constant to another variable. •It is applied to individual units of a program. •Software developers perform structural testing on the individual program units they write.