ISTQB 1.4 - Test Process

Test Planning (Test Process) (K)

*The activity of establishing or updating a test plan.* Test planning involves activities that define - the objectives of testing and - the approach for meeting test objectives within constraints imposed by the context (e.g., specifying suitable test techniques and tasks, and formulating a test schedule for meeting a deadline). Test plans may be revisited based on feedback from monitoring and control activities.

Test Analysis (Test Process) (K)

*The activity that identifies test conditions by analyzing the test basis.* During test analysis, - the test basis is analyzed to identify testable features and define associated test conditions. - In other words, test analysis determines *"what to test"* in terms of measurable coverage criteria.

The evaluation of exit criteria for test execution as part of a given test level may include:

- Checking test results and logs against specified coverage criteria - Assessing the level of component or system quality based on test results and logs - Determining if more tests are needed (e.g., if tests originally intended to achieve a certain level of product risk coverage failed to do so, requiring additional tests to be written and executed) (i.e. did our tests meet the objectives.)

Test Completion Activities

- Checking whether all defect reports are closed, entering change requests or product backlog items for any defects that remain unresolved at the end of test execution - Creating a test summary report to be communicated to stakeholders - Finalizing and archiving the test environment, the test data, the test infrastructure, and other testware for later reuse - Handing over the testware to the maintenance teams, other project teams, and/or other stakeholders who could benefit from its use - Analyzing lessons learned from the completed test activities to determine changes needed for future iterations, releases, and projects - Using the information gathered to improve test process maturity

Test Implementation Activities

- Developing and prioritizing test procedures, and, potentially, creating automated test scripts - Creating test suites from the test procedures and (if any) automated test scripts. - Arranging the test suites within a test execution schedule in a way that results in efficient test execution - Building the test environment (including, potentially, test harnesses, service virtualization, simulators, and other infrastructure items) and verifying that everything needed has been set up correctly - Preparing test data and ensuring it is properly loaded in the test environment - Verifying and updating bi-directional traceability between the test basis, test conditions, test cases, test procedures, and test suites

Test Analysis - In some cases...

- In some cases, it produces test conditions which are to be used as test objectives in test charters

Test Product Reports

- Produced on a regular basis

Test Execution Activities

- Recording the IDs and versions of the test item(s) or test object, test tool(s), and testware - Executing tests either manually or by using test execution tools - Comparing actual results with expected results - Analyzing anomalies to establish their likely causes (e.g., failures may occur due to defects in the code, but false positives also may occur. - Reporting defects based on the failures observed - Logging the outcome of test execution (e.g., pass, fail, blocked) - Repeating test activities either as a result of action taken for an anomaly, or as part of the planned testing (e.g., execution of a corrected test, confirmation testing, and/or regression testing) - Verifying and updating bi-directional trace

Contextual factors that influence the test process for an organization

- Software development lifecycle model and project methodologies being used. - Test levels and test types being considered. - Product and project risks - Business domain - Operational constraints, including but not limited to: > Budget and resources > Timescales > Complexity > Contractual and regulatory requirements. - Organizational policies and practices - Required internal and external standards.

General aspects of the test process

- Test activities and tasks - Test work products - Traceability between the test basis and test work products.

Test Activities

- Test planning - Test monitoring and control - Test analysis - Test design - Test implementation - Test execution - Test completion

FL-1.4.2 (K2) Describe the test activities and respective tasks within the test process (O)

- Test planning The activity of establishing or updating a test plan. - Test monitoring and control Test monitoring - comparison of actual progress against the test plan Test control - taking actions necessary to meet the objectives of the test plan Both - supported by the evaluation of exit criteria - Test analysis Determines *"what to test"* in terms of measurable coverage criteria - Test design Answers the question *"how to test?"* - Test implementation "do we now have everything in place to run the test?" - Test execution The process of running a test on the component or system under test, producing actual result(s). - Test completion The activity that makes test assets available for later use, leaves test environments in a satisfactory condition and communicates the results of testing to relevant stakeholders.

Test Product Creation

- Which test activities are involved in this test process? - How these activities are implemented? - When do these activities occur? - This may be discussed during an organization's test strategy.

Identification of defects during test analysis

- is an important test benefit especially where no other review process is being used and/or the test process is closely connected with the review process. - Such test analysis activities not only verify whether the requirements are consistent, properly expressed, and complete, but also validate whether the requirements properly capture customer, user, and other stakeholder needs. - (e.g.) techniques such as behavior driven development (BDD) and acceptance test driven development (ATDD), which involve generating test conditions and test cases from user stories and acceptance criteria prior to coding, also verify, validate, and detect defects in the user stories and acceptance criteria

Test Analysis Activities

1. Analyzing the test basis appropriate to the test level being considered 2. Evaluating the test basis and test items to identify defects of various types 3. Identifying features and sets of features to be tested 4. Defining and prioritizing test conditions for each feature based on analysis of the test basis, and considering functional, non-functional, and structural characteristics, other business and technical factors, and levels of risks. 5. Capturing bi-directional traceability between each element of the test basis and the associated test conditions

Test Analysis Activity 1

1. Analyzing the test basis appropriate to the test level being considered Examples - Requirement specifications, such as business requirements, functional requirements, system requirements, user stories, epics, use cases, or similar work products that specify desired functional and non-functional component or system behavior. - Design and implementation information, such as system or software architecture diagrams or documents, design specifications, call flows, modelling diagrams (e.g., UML or entity-relationship diagrams), interface specifications, or similar work products that specify component or system structure. - The implementation of the component or system itself, including code, database metadata and queries, and interfaces. - Risk analysis reports, which may consider functional, non-functional, and structural aspects of the component or system.

Test Design Activities

1. Designing and prioritizing test cases and sets of test cases 2. Identifying necessary test data to support test conditions and test cases 3. Designing the test environment and identifying any required infrastructure and tools 4. Capturing bi-directional traceability between the test basis, test conditions, test cases, and test procedures

Test Analysis Activity 2

2. Evaluating the test basis and test items to identify defects of various types, such as: - Ambiguities - Omissions - Inconsistencies - Inaccuracies - Contradictions - Superfluous statements

Test Analysis Activity 3

3. Identifying features and sets of features to be tested.

Test Analysis Activity 4

4. Defining and prioritizing test conditions for each feature based on analysis of the test basis, and considering functional, non-functional, and structural characteristics, other business and technical factors, and levels of risks.

Test Analysis Activity 5

5. Capturing bi-directional traceability between each element of the test basis and the associated test conditions (- The requirements are traceable to the test conditions and the test conditions are traceable to the requirements.)

Risk

A factor that could result in future negative consequences.

Test Type

A group of test activities based on specific test objectives aimed at specific characteristics of a component or system.

Test Schedule

A list of activities, tasks or events of the test process, identifying their intended start and finish dates and/or times, and interdependencies.

Milestone

A point in time in a project at which defined (intermediate) deliverables and results should be ready.

Test Objective (K)

A reason or purpose for designing and executing a test.

Non-Functional Requirement

A requirement that describes how the component or system will do what it is intended to do.

Functional Requirement

A requirement that specifies a function that a component or system must be able to perform.

Test Case (K)

A set of preconditions, inputs, actions (where applicable), expected results and postconditions, developed based on test conditions.

Blocked Test Case

A test case that cannot be executed because the preconditions for its execution are not fulfilled.

Fail

A test is deemed to fail if its actual result does not match its expected result.

Pass

A test is deemed to pass if its actual result matches its expected result.

Test Management Tool

A tool that provides support to the test management and control part of a test process. It often has several capabilities, such as testware management, scheduling of tests, the logging of results, progress tracking, incident management and test reporting.

Are activity groups logically sequential?

Although many of these activity groups may appear logically sequential, they are often implemented iteratively.

Exploratory Testing

An approach to testing whereby the testers dynamically design and execute tests based on their knowledge, exploration of the test item and the results of previous tests.

Test Condition (K)

An aspect of the test basis that is relevant in order to achieve specific test objectives. (What to test)

Iteration - Iteratively

An iteration is a complete development loop resulting in a release (internal or external) of an executable product, a subset of the final product under development, which grows from iteration to iteration to become the final product.

Test Data (K)

Data created or selected to satisfy the execution preconditions and inputs to execute one or more test cases.

Coverage Criteria

Did we cover this test basis or not?

Test Plan

Documentation describing the - test objectives to be achieved and - the means and the schedule for achieving them, organized to coordinate testing activities.

Test Report

Documentation summarizing test activities and results.

FL-1.4.1 (K2) Explain the impact of context on the test process. (O)

Each individual product will come with it own requirements. You have to look at the context of the product to develop a good test process. It is very useful that the test basis (for any level or type of testing that is being considered) has measurable coverage criteria defined. Coverage criteria can act effectively as key performances indicators (KPIs) to drive the activities that demonstrate achievement of software test objectives.

Definition of Done (DoD)

Exit criteria

Acceptance Testing

Ref: After ISO 24765 See Also: user acceptance testing Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.

Test Tool

See Also: CAST A software product that supports one or more test activities, such as planning and control, specification, building initial files and data, test execution and test analysis.

Anomaly

See Also: defect, error, fault, failure, incident, problem Any condition that deviates from expectation based on requirements specifications, design documents, user documents, standards, etc., or from someone's perception or experience. Anomalies may be found during, but not limited to, reviewing, testing, analysis, compilation, or use of software products or applicable documentation.

Test Charter

See Also: exploratory testing Synonyms: charter Documentation of test activities in session-based exploratory testing. - typical work products in some types of experience-based testing. - a document that we use in experience-based testing to write our test ideas. - in the test charter you write test conditions as test objectives that you need to cover in your testing. Test charters are typical work products in some types of experience-based testing. When these test objectives are traceable to the test basis, coverage achieved during such experience-based testing can be measured.

Low-level test case

See Also: high-level test case Synonyms: concrete test case A test case with concrete values for input data and expected results.

Defect Management Tool

See Also: incident management tool Synonyms: bug tracking tool , defect tracking tool A tool that facilitates the recording and status tracking of defects.

High-level Test Case

See Also: low-level test case Synonyms: abstract test case , logical test case A test case without concrete values for input data and expected results.

Product Risk

See Also: risk A risk impacting the quality of a product.

Project Risk

See Also: risk A risk that impacts project success.

Test Design (Test Process) (K)

See Also: test design specification The activity of deriving and specifying test cases from test conditions. - During test design, the test conditions are elaborated into high-level test cases, sets of high-level test cases, and other testware. - test design answers the question *"how to test?"*

Test Monitoring (K)

See Also: test management *A test management activity that involves checking the status of testing activities, identifying any variances from the planned or expected status, and reporting status to stakeholders.* Test monitoring involves the on-going comparison of actual progress against the test plan using any test monitoring metrics defined in the test plan.

Test Control (K)

See Also: test management A test management task that deals with developing and applying a set of corrective actions to get a test project on track when monitoring shows a deviation from what was planned. Test control involves taking actions necessary to meet the objectives of the test plan (which may be updated over time).

Test Script

See Also: test procedure A sequence of instructions for the execution of a test.

Test Procedures (K)

See Also: test script A sequence of test cases in execution order, and any associated actions that may be required to set up the initial preconditions and any wrap up activities post execution. - The sequence of the test cases. - How will we execute it?

Black-box Test Technique

Synonyms: black-box technique , specification-based technique , specification-based test technique A procedure to derive and/or select test cases based on an analysis of the specification, either functional or non-functional, of a component or system without reference to its internal structure. (Provide input to the system and wait for an output.)

Exit Criteria

Synonyms: completion criteria , test completion criteria , definition of done The set of conditions for officially completing a defined task. (- Goals that we must achieve. - Tells us if our test plans are satisfied or not.)

Experience-based Test Techniques

Synonyms: experience-based technique A procedure to derive and/or select test cases based on the tester's experience, knowledge and intuition. - means that you are an experienced tester and are using you experience as a guide to testing.

Performance Indicator aka Key Performance Indicator (KPI)

Synonyms: key performance indicator A high-level metric of effectiveness and/or efficiency used to guide and control progressive development, e.g., lead-time slip for software development.

Test Oracle (K)

Synonyms: oracle A source to determine expected results to compare with the actual result of the system under test.

Test Strategy

Synonyms: organizational test strategy Documentation that expresses the generic requirements for testing one or more projects run within an organization, providing detail on how testing is to be performed, and is aligned with the test policy.

White-box Test Technique

Synonyms: structural test technique , structure-based test technique , structure-based technique , white-box technique A procedure to derive and/or select test cases based on an analysis of the internal structure of a component or system. (Provide input, then look at the code, then wait for the output.)

Test Environment

Synonyms: test bed , test rig An environment containing hardware, instrumentation, simulators, software tools, and other support elements needed to conduct a test.

Test Technique

Synonyms: test case design technique , test specification technique , test technique , test design technique A procedure used to derive and/or select test cases.

Test Suite (K)

Synonyms: test case suite , test set A set of test cases or test procedures to be executed in a specific test cycle.

Coverage (K)

Synonyms: test coverage The degree to which specified coverage items have been determined or have been exercised by a test suite expressed as a percentage.

Test Log

Synonyms: test record , test run log A chronological record of relevant details about the execution of tests.

Test Summary Report

Synonyms: test report A test report that provides an evaluation of the corresponding test items against exit criteria. (Produced at completion milestones)

Test Level

Synonyms: test stage A specific instantiation of a test process.

Test Progress Report

Synonyms: test status report A test report produced at regular intervals about the progress of test activities against a baseline, risks, and alternatives requiring a decision. (- Reports on the progress of the test for the stakeholders. - including what we did, deviations from the plan and information to support any decision to stop testing.)

Test Analysis Work Products

Test analysis work products include defined and prioritized test conditions, each of which is ideally bi-directionally traceable to the specific element(s) of the test basis it covers. For exploratory testing, test analysis may involve the creation of test charters. Test analysis may also result in the discovery and reporting of defects in the test basis.

Test Completion Work Products

Test completion work products include test summary reports, action items for improvement of subsequent projects or iterations (e.g., following a project Agile retrospective), change requests or product backlog items, and finalized testware.

Test Design Work Products

Test design results in test cases and sets of test cases to exercise the test conditions defined in test analysis. It is often a good practice to design high-level test cases, without concrete values for input data and expected results. Such high-level test cases are reusable across multiple test cycles with different concrete data, while still adequately documenting the scope of the test case. Ideally, each test case is bidirectionally traceable to the test condition(s) it covers. Test design also results in the design and/or identification of the necessary test data, the design of the test environment, and the identification of infrastructure and tools, though the extent to which these results are documented varies significantly. Test conditions defined in test analysis may be further refined in test design.

Test Execution Work Products

Test execution work products include: - Documentation of the status of individual test cases or test procedures (e.g., ready to run, pass, fail, blocked, deliberately skipped, etc.) - Defect reports - Documentation about which test item(s), test object(s), test tools, and testware were involved in the testing Ideally, once test execution is complete, the status of each element of the test basis can be determined and reported via bi-directional traceability with the associated the test procedure(s). For example, we can say which requirements have passed all planned tests, which requirements have failed tests and/or have defects associated with them, and which requirements have planned tests still waiting to be run. This enables verification that the coverage criteria have been met, and enables the reporting of test results in terms that are understandable to stakeholders.

Test Implementation Work Products

Test implementation work products include: - Test procedures and the sequencing of those test procedures - Test suites - A test execution schedule Ideally, once test implementation is complete, achievement of coverage criteria established in the test plan can be demonstrated via bi-directional traceability between test procedures and specific elements of the test basis, through the test cases and test conditions. In some cases, test implementation involves creating work products using or used by tools, such as service virtualization and automated test scripts. Test implementation also may result in the creation and verification of test data and the test environment. The completeness of the documentation of the data and/or environment verification results may vary significantly. The test data serve to assign concrete values to the inputs and expected results of test cases. Such concrete values, together with explicit directions about the use of the concrete values, turn high-level test cases into executable low-level test cases. The same high-level test case may use different test data when executed on different releases of the test object. The concrete expected results which are associated with concrete test data are identified by using a test oracle. In exploratory testing, some test design and implementation work products may be created during test execution, though the extent to which exploratory tests (and their traceability to specific elements of the test basis) are documented may vary significantly. Test conditions defined in test analysis may be further refined in test implementation.

Test Monitoring and Control Work Products

Test monitoring and control work products typically include various types of test reports, including test progress reports (produced on an ongoing and/or a regular basis) and test summary reports (produced at various completion milestones). All test reports should provide audience-relevant details about the test progress as of the date of the report, including summarizing the test execution results once those become available. Test monitoring and control work products should also address project management concerns, such as task completion, resource allocation and usage, and effort.

Test Monitoring and Control (Test Process)

Test monitoring involves the on-going comparison of actual progress against the test plan using any test monitoring metrics defined in the test plan. Test control involves taking actions necessary to meet the objectives of the test plan (which may be updated over time). Test monitoring and control are supported by the evaluation of exit criteria, which are referred to as the definition of done in some lifecycles. Test progress against the plan is communicated to stakeholders in test progress reports, including deviations from the plan and information to support any decision to stop testing.

Test Planning Work Products

Test planning work products typically include one or more test plans. The test plan includes information about the test basis, to which the other test work products will be related via traceability information (see below and section 1.4.4), as well as exit criteria (or definition of done) which will be used during test monitoring and control. Test plans are described in section 5.2.

Test Work Products

Test work products are created as part of the test process. Just as there is significant variation in the way that organizations implement the test process, there is also significant variation in the types of work products created during that process, in the ways those work products are organized and managed, and in the names used for those work products. Many of the test work products described in this section can be captured and managed using test management tools and defect management tools

Software Development Lifecycle

The activities performed at each stage in software development, and how they relate to one another logically and chronologically.

Test Completion (Test Process) (K)

The activity that makes test assets available for later use, leaves test environments in a satisfactory condition and communicates the results of testing to relevant stakeholders. Test completion activities collect data from completed test activities to consolidate experience, testware, and any other relevant information. Test completion activities occur at project milestones such as when a software system is released, a test project is completed (or cancelled), an Agile project iteration is finished (e.g., as part of a retrospective meeting), a test level is completed, or a maintenance release has been completed.

Test Implementation (K)

The activity that prepares the testware needed for test execution based on test analysis and design.

Test Implementation (Test Process) (K)

The activity that prepares the testware needed for test execution based on test analysis and design. During test implementation, the testware necessary for test execution is created and/or completed, including sequencing the test cases into test procedures. - "do we now have everything in place to run the test?"

Test Analysis Testing Techniques

The application of - black-box, - white-box, and - experience-based tests They can be useful in the process of test analysis to reduce the likelihood of omitting important test conditions and to define more precise and accurate test conditions.

Test Basis (K)

The body of knowledge used as the basis for test analysis and design. Things that we need for testing; like - requirements - user stories

Testability

The degree of effectiveness and efficiency with which tests can be designed and executed for a component or system.

Test Design...

The elaboration of test conditions into test cases and sets of test cases during test design often involves using test techniques. - As with test analysis, test design may also result in the identifications of similar types of defects in the test basis - As with test analysis, the identification of defects during test design is an important potential benefit.

Test Infrastructure

The organizational artifacts needed to perform testing, consisting of test environments, test tools, office environment and procedures.

Risk Analysis

The overall process of risk identification and risk assessment.

Test Management

The planning, scheduling, estimating, monitoring, reporting, control and completion of test activities.

Maintenance

The process of modifying a component or system after delivery to correct defects, improve quality attributes, or adapt to a changed environment.

Test Execution (Test Process) (K)

The process of running a test on the component or system under test, producing actual result(s). During test execution, test suites are run in accordance with the test execution schedule.

Behavior

The response of a component or system to a set of input values and preconditions.

Test Process

The set of interrelated activities comprising of - test planning, - test monitoring and control, - test analysis, - test design, - test implementation, - test execution, and - test completion. Common sets of test activities without which testing will be less likely to achieve its established objectives Set of test activities that are generally used when a tester tests.

Testware

Work products produced during the test process for use in planning, designing, executing, evaluating and reporting on testing. (Any software that deals with testing.)

Test design and test implementation tasks...

are often combined.

Exploratory testing may be based on...

test charters (produced as part of test analysis), and exploratory tests are executed immediately as they are designed and implemented

In exploratory testing and other types of experience-based testing...

test design and implementation may occur, and may be documented, as part of test execution.