CIS 3374 IT Quality Assurance & Testing (Midterm FA18)

Agile Model

"Agile Development" is an umbrella term for several iterative software development methodologies. It is a very flexible model the main features of which are: - requirements and priorities are dynamic - there is an option of making changes in the short period of time - iterative approach - Both development and testing activities present at the same time

User Story Format

"As a <role>,I want <goal/desire>[so that <benefit>]"

Non-Functional Requirements

"How well system does it" "system shall be <requirement>" - an overall property of the system as a whole or of a particular aspect and not a specific function. Usually detailed statements of the conditions under which the solution must remain effective, qualities that the solution must have, or constraints within which it must operate. Examples include: reliability, testability, maintainability, availability. Also known as the ilities.

Testing Methods

Static testing, dynamic testing, white box testing, black box testing.

Testing Types

Test Automation, Exploratory Testing, Regression Testing.

Software Testing Documentation

Test Plan; Test Cases; Test Suite; Test Strategy; Traceability Matrix; Test Script; etc.

What test cases should you not automate?

Test cases that are newly designed and not executed manually at least once, if the requirements are changing frequently, executed random basis.

Identify at least 2 characteristics of the Agile testing mindset.

Testing is an activity not a phase Prevent bugs rather than finding bugs Don't be a checker, be a tester Don't try to break the system, instead help build the best possible system The whole team is responsible for quality, not just the tester

Stand-Up Meeting

A short, daily all-hands meeting in which members of an Agile team address three key questions: • What did you get done since the last stand-up? • What will you do before the next stand-up? • What impediments stand in your way?

Retrospective

A time boxed meeting held at the end of an iteration, or at the end of a release, in which the team examines its processes to determine what succeeded and what could be improved.

Bug Life-Cycle

Bug found; Bug gets reported; It goes to Development Manager to get Assigned To and Priority; Developer sees the report, fixes the bug, marks it as Fixed; It goes to Tester for verification of the fix; If it is fixed Tester close the bug.

Requirements Hierarchy

Business Requirements at top (High Level) User Requirements in middle (Mid level) System Requirements on bottom of pyramid (Detailed)

Agile Testing Definition

Collaborative testing practices that occur continuously, from inception to delivery and beyond

Test Charter Format

Explore (target), With (resources), To discover (Information)

SQA Challenges

High complexity Diverse and potentially very large number of users Invisibility of the product Limited opportunities to detect defects ("bugs")

User Requirements

Mid-level statements of the needs of a particular stakeholder or group of stakeholders. They usually describe how someone wants to interact with the intended solution.

Burn Down Chart

Plots units of work that remain to accomplish (Y axis) against units of time (X axis).

Product Owner

On an Agile development team, the real customer or end user, or a stand-in for the customer or end user; a non-coding team member who has a complete grasp of the requirements and business value of the product and is responsible for prioritizing work for the team.

Agile Testing Matrix

Q1 (Bottom Left) - Automated: Unit Tests, Component Tests. Q2 (Top Left) - Automated & Manual: Functional Tests, Examples, Story Tests, Prototypes, Simulations. Q3 (Top Right) - Manual: Exploratory Testing, Scenarios, Usability Testing, User Acceptance Testing Q4 (Bottom Right) - Tools: Security testing, "ility" testing, Performance & Load testing

Requirements Metrics

Quantity of use cases, requirements, etc. Volatility refers to additions, deletions and modifications of requirements during the systems development life cycle.

Test Charter Definition

a goal or agenda for a test session. It defines the target to explore and provides direction but not specific actions.

Functional Requirements

"What system does" "system shall do <requirement>" -an individual action or part of the system. Usually detailed statements of capabilities, behavior, and information that the solution will need.

Software Quality

(1)The degree to which a system, component, or process meets specified requirements. (2)The degree to which a system, component, or process meets customer or user needs or expectations.

SQA Objectives

1. Assuring an acceptable level of confidence that the software will conform to functional and technical requirements. 2. Assuring an acceptable level of confidence that the software will conform to managerial scheduling and budgetary requirements.

Graphical user interface testing

A testing framework generates user interface events such as keystrokes and mouse clicks, and observes the changes that result in the user interface, to validate that the observable behavior of the program is correct.

API driven testing

A testing framework that uses a programming interface to the application to validate the behavior under test. Typically API driven testing bypasses application user interface altogether. It can also be testing public (usually) interfaces to classes, modules or libraries.

Testing Definition

An empirical, technical investigation conducted to provide stakeholders with information about the quality of the product or service under test. Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc.

Backlog

An ever-evolving list of product requirements, prioritized by the customer (or customer representative), that conveys to an Agile team which features to implement first.

Requirement Types

Architectural, Business, user, functional, non-functional

Regression Testing

Regression testing focuses on finding defects after a code change has occurred.

Quality Triangle for Waterfall

Scope Cost Schedule Scope is fixed (top point of triangle), Cost and Schedule are variable (at bottom of triangle)

Exploratory Test Sessions

Sessions are planned uninterrupted, chartered, and reviewable Time-based (45, 90, 120) Notes documenting the paths taken and results uncovered

INVEST

Characteristics of a good user story Independent Negotiable Valuable Estimable Small Testable **which is like "Verifiable" from a good requirement**

Software

Computer programs, procedures, and possibly associated documentation and data pertaining to the operation of a computer system.

Quality Triangle for Agile

Cost Schedule Scope Cost and schedule are fixed (top of triangle), scope is variable (bottom point of triangle).

User Story Definition

Definition: An Agile requirement, stated as a sentence or two of plain English. A user story is often expressed from the user's point of view, and describes a unit of desired functionality. User stories are used as the basis for defining the functions a business system must provide. It captures the 'who', 'what' and 'why' of a requirement in a simple, concise way. Every user story must at some point have one or more acceptance tests, allowing the developer to test when the user story is done.

Validation

Did we build the right software product? The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements

Verification

Did we build the software product Correctly? The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase

Boehm's First Law

Errors are more frequent during requirements and design activities and are more expensive the later they are removed.

Architectural Requirements

Explain what has to be done by identifying the necessary systems structure and systems behavior, i.e., architecture of a system.

Acceptance Tests

Functional testing of a user story by the agile team Acceptance tests are black-box tests. A story can have one or many acceptance tests, whatever it takes to ensure the functionality works. A user story is not considered complete until it has passed its acceptance tests. This means that new acceptance tests must be created for each iteration or the development team will report zero progress! State an intent, not a solution (e.g., "The user can choose an account" rather than "The user can select the account from a drop-down")

Acceptance Criteria Syntax

Given (starting point), when (action), then (outcome)

Business Requirements

High-level statements of the goals, objectives, or needs of an organization. They usually describe opportunities that an organization wants to realize or problems that they want to solve.

Static Testing

Item under test NOT being executed. Involves verification, reviews, walk-through, inspections.

Dynamic Testing

Item under test is being executed. Validation, exploratory, etc.

Iteration or Sprint

The uninterrupted period of time during which an Agile development team performs work, most commonly one week to one month in length, at the end of which the team delivers "potentially shippable" product.

Black Box Testing

Think of a book-bag that you don't know what is inside it. Examining functionality without any knowledge of internal implementation

Releases

This is a process of delivering and providing the product to clients

Testing Levels

Unit Test: test at lowest & most specific level Integration Test: test units together (modules) System Test: test entire system User Acceptance Test: tests that evaluate the system's compliance with the business requirements and assess whether it is acceptable for delivery.

Identify and define at least the 3 levels of testing, as shown on the right hand side of the V-Model

Unit Testing refers to tests that verify the functionality of a specific section of code. Integration Testing is any type of software testing that seeks to verify the interfaces between components against a software design. System Testing tests a completely integrated system to verify that it meets its requirements. Acceptance Testing refers to tests that evaluate the system's compliance with the business requirements and assess whether it is acceptable for delivery.

Characteristics of a Good Requirement

Unitary (Cohesive), Complete, Consistent, Non-Conjugated (Atomic), Traceable, Current, Unambiguous, Verifiable

V Model Definition

Verification on left side, Validation on right side. SDLC model where execution of processes happens in a sequential manner in a V-shape. It is also known as Verification and Validation model. The V-Model is an extension of the waterfall model and is based on the association of a testing phase for each corresponding development stage. This means that for every single phase in the development cycle, there is a directly associated testing phase. This is a highly-disciplined model and the next phase starts only after completion of the previous phase.

V Model Components

Verification on left side: Business Requirement -> Acceptance Test Plan Software Application -> System Test Plan High Level Design -> Integration Test Plan Detailed Design -> Unit Test Plan When designing each of these parts the test plans can be created during the process to be used during validation (right side).

What is primary measure of progress on an agile development project?

Working or shippable software, or minimally viable product (MVP)

Design Review

a systematic, comprehensive, and documented analysis of a design to determine its capability and adequacy to meet its requirements. A design review also serves to identify present and potential problems.

Agile Advantages

• Customer satisfaction by rapid, continuous delivery of useful software. • People and interactions are emphasized rather than process and tools. Customers, developers and testers constantly interact with each other. • Working software is delivered frequently (weeks rather than months). • Face-to-face conversation is the best form of communication. • Close, daily cooperation between business people and developers. • Continuous attention to technical excellence and good design.

Prototyping Disadvantages

• Implementing and then repairing way of building systems. • May increase the complexity of the system as scope of the system may expand beyond original plans. • Incomplete application may cause application not to be used as the full system was designed.

Agile Disadvantages

• In case of some software deliverables, especially the large ones, it is difficult to assess the effort required at the beginning of the software development life cycle. • There is a lack of emphasis on necessary designing and documentation. • The project can easily get taken off track if the customer representative is not clear what final outcome that they want

Prototyping Disadvantages

• Leads to implementing and then repairing way of building systems. • May increase the complexity of the system as scope of the system may expand beyond original plans. • Incomplete application may cause application not to be used as the full system was designed.

Waterfall Disadvantages

• Once an application is in the testing stage, it is very difficult to go back and change something. • No working software is produced until late during the life cycle. • High amounts of risk and uncertainty. • Not a good model for complex and object-oriented projects, or for long and on-going projects. • Not suitable for the projects where requirements are at a moderate to high risk of changing.

Waterfall Advantages

• This model is simple and easy to understand and use. • It is easy to manage due to the rigidity of the model -each phase has specific deliverables and a review process. • In this model phases are processed and completed one at a time. Phases do not overlap. • Waterfall model works well for smaller projects where requirements are very well understood.

Prototyping Advantages

• Users are actively involved in the development. • Since a working model of the system is provided, the users get a better understanding of the system being developed. • Errors can be detected much earlier. • Quicker user feedback is available leading to better solutions. • Missing functionality can be identified easily. • Confusing or difficult functions can be identified.

Why Automate?

•Saves time and money by making testing more efficient •Improves testing accuracy compared to testing directed by humans •Increases test coverage because multiple testing tools can be deployed at once allowing for parallel testing of different test scenarios •Helps developers by finding bugs and errors more quickly

Agile Development Process

• Gather & Write Requirements • Create User Stories with Acceptance Criteria • Write Acceptance Tests • Code • Write Unit Tests • Execute Tests and Report Results • Report a Defect • Fix Defect & Rerun Tests

Prototyping Definition

type of development in which emphasis is placed on developing prototypes early in the development process to permit early feedback and analysis in support of the development process.

Minimum Viable Product (MVP)

A product with just enough features to satisfy early customers, and to provide feedback for future product development.

Requirements Traceability

the ability to describe and follow the life of a requirement, in both forwards and backwards direction

Software quality assurance (SQA)

1.A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements. 2.A set of activities designed to evaluate the process by which the products are developed or manufactured.

Test Automation Strategy

70% unit tests (bottom tier of pyramid), 20% API tests (middle of pyramid), 10% GUI Tests, Exploratory testing at tip of pyramid (costs the most money and takes most time).

Done

A Team's universally agreed-upon criteria for what makes a unit of work "potentially shippable."

Willa program defect that a team finds during analysis be more or less expensive to fix than if the defect was found during testing? Why?

A defect is less expensive to fix in the earlier stages of development like analysis than later in testing, because less time is spent investigating the cause of the defect and there are less project deliverables to change.

Is a developer more likely to perform black-box or white-box testing? Why?

A developer is more likely to perform white-box testing, as the developer knows and tests the internal structures or workings of a program.

Persona

A fictional character with individual needs, goals and habits, created by an Agile team as a representative user, to serve as a reference-point for usability during product development.

Cost of Quality

A means to quantify the total cost of quality-related efforts and deficiencies.

Sprint Review or Demo

A meeting, held at the end of an iteration, at which the development team demonstrates working software and solicits feedback from the product owner, the customer, management, other development teams and other project stakeholders

Agile Manifesto

Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan

Peer Reviews

Inspections -a means of verifying products by manually examining the developing product, a piece at a time, by small groups of peers to ensure that it is correct and conforms to product specifications and requirements. • Walkthroughs-a form of software peer review in which a designer or programmer leads members of the development team and other interested parties through a software product, and the participants ask questions and make comments about possible errors, violation of development standards, and other problems.

Requirements Definition

Software Requirements deal with establishing the needs of stakeholders that are to be solved by software. •A condition or capability needed by a user to solve a problem or achieve an objective •A condition or capability that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed document.

If members of a team conduct a walk-through of a program, has the team performed static or dynamic testing? Why?

Static Testing -a form of software testing where the actual program is not used. ◦Examples: Reviews, walk-through's, inspections Dynamic testing takes place when the program itself is run by actually executing a given set of test cases.A walk-through is considered static testing, as the product or program is not being tested

Project Management Triangle (Quality Triangle)

TIME COST SCOPE. model of the constraints of project management. It is used to illustrate that project management success is measured by the project team's ability to manage the project, so that the expected results are produced while managing time and cost.

What are the primary responsibilities of a Product Owner on an agile development team?

The Product Owner is a stand-in for the customer or end user; a non-coding team member who has a complete grasp of the requirements and business value of the product and is responsible for prioritizing work for the team.

Scrum Master

The ScrumMaster's role is more that of coach and facilitator, a role that sits between the project and the customer. The ScrumMaster doesn't manage the team that produces the work; instead, he supports the product owner, coaches the team, and makes sure that Scrum processes are adhered to. The ScrumMaster is responsible for the Scrum process, and its correct and continuous implementation.

User Story Creation

The Three C's formula -Card, Conversation, Confirmation -capture the components of a user story -A Card is a physical token giving tangible and durable form to what would only be an abstraction; -A Conversation is taking place at different time and places during a project between the various stakeholders concerned by the given feature (customers, users, developers, testers, etc.), which is largely verbal but most often supplemented by documentation; -The Confirmation, the more formal the better, ensures that the objectives the conversation revolved around have been reached finally.

Identify two types of tests that are in Q3 of the Agile Testing Matrix? What are these tests used for?

The following tests are in Q3 of the Agile Testing Matrix: Exploratory, Usability, User Acceptance, Alpha/Beta Tests.Also black box tests. These tests are used to critique/test the product and can be used by the customer/business to test a product.

What Test Cases To Automate?

The ones that are executed repeatedly, tedious to perform manually, time consuming, business critical.

Acceptance Criteria

are the conditions that a software product must satisfy to be accepted by a user, customer, or in the case of system level functionality, the consuming system. set of statements, each with a clear pass/fail result, that specify both functional and non-functional requirements, and are applicable at the Epic, Feature, and Story Level. Acceptance criteria constitute our "Definition of Done".

An agile development is in the last week of a sprint. The scrum master remarks: "We are not going to be able to finish all features by the end of the sprint." What should the team do? Why?

b.Finish what they can during the sprint and add the unfinished features to the product backlog. Scope is the constraint that can be changed in the quality triangle for agile projects.

Waterfall model

through the phases of 1) Planning (Planning & Concept development) 2) Analysis (Analysis & Requirements gathering) 3) Design (Architecture & Specifications) 4) Development (Development & Test) 5) Implementation/Release 6) Maintenance Waterfall doesn't allow review or come back to any prior phase once it's complete. In "Waterfall model" phases are executed sequentially.

Test Automation

use of special software to control the execution of tests and the comparison of actual outcomes with predicted outcomes.

SDLC

describes a process for planning, creating, testing, and deploying an information system (or product).

SHALL

indicate mandatory requirements

software requirements specification(SRS)

is a description of a software system to be developed.•Establishes the basis for an agreement between customers and suppliers on what the software product is to do as well as what it is not expected to do.

Project Plan

is a formal document to summarize business, management and financial aspects of a project. "Contract" between Project Manager and customers. It includes scope, objectives, benefits, costs, risks and plans, etc.

Model Based Testing

is a testing technique where run time behavior of a software under test is checked against predictions made by a formal specification or model. In other means, it describes how system behaves in response to an action (determined by a model).

Exploratory Testing

is about exploring, finding out about the software, what it does, what it doesn't do, what works and what doesn't work. ◦The tester is constantly making decisions about what to test next and where to spend the (limited) time. ◦Exploratory testing is a hands-on approach in which testers are involved in minimum planning and maximum test execution.

Session Based Test Management

method for measuring and managing exploratory testing

White Box Testing

tests internal structures or workings of a program, as opposed to the functionality exposed to the end-user.