Area 3 - Testing and SCM

What are the pros and cons of the git workflow?

+Always stable code in the master +Can work on release and future features simultaneously +Ability to fix issues without disrupting workflow -Not good for supporting multiple product versions -History can get complicated -Many merging conflicts to solve

What are the pros and cons of the feature branches workflow?

+Easier conflict resolution +Better control of what features to include when +Possibility of using pull requests -Additional steps -Long living branches might be hard to merge

What are the pros and cons of a centralized workflow?

+Little setup +Will work for small projects -Limited flexibility -Hard to maintain a continuously working product

What are the pros and cons of a decentralized workflow?

+No need to provide write access to main repository +Good for large open-source projects -More complicated to set up and maintain

What is a smoke test?

-A smoke test is a preliminary testing to reveal simple failures severe enough to, for example, reject a prospective software release. -The purpose of smoke testing is to determine whether the application is so badly broken that further immediate testing is unnecessary. -Smoke tests frequently run quickly, giving benefits of faster feedback, rather than running more extensive test suites, which would naturally take much longer.

Which are the two different types of pilot tests?

-Alpha test: at the developer's site, controlled environment -Beta test: at one or more customer sites Testing the system in-house (alpha) and then beta-testing it on a selected group of people at the customer's site, selected in order to represent all kinds of system usage, is a common approach when systems are to be released to a wide variety of customers.

Which questions do we ask when testing software?

-Are the functions giving correct output? -Are the integrated modules giving correct output? -Is the entire system giving correct output when used? -Is the correct output given in reasonable time? -Is the output presented in an understandable way? -Was this what we really expected?

Which are the three different types of acceptance tests?

-Benchmark test -Pilot test -Parallel test

Which factors influence the decision to stop testing?

-Deadlines, e.g., release deadlines, testing deadlines -Test cases are completed with certain percentage passed -The test budget has been depleted -Coverage of code, functionality, or requirements reaches a specified point

Which three types of black box testing are there?

-Exhaustive testing -Equivalence class testing (aka equivalence partitioning) -Boundary value analysis

Which other methods for validation and verification are there?

-Formal verification -Model checking -Prototyping -Simulation -Software reviews

How do you identify the test cases?

1.Assign a unique number to each EC. 2.Until all valid ECs have been covered by test cases, write a new test case covering as many of the uncovered valid ECs as possible. 3.Until all invalid ECs have been covered by test cases, write a test case that cover one, and only one, of the uncovered invalid ECs.

Which are the four integration testing techniques?

1.Big-bang 2.Bottom-up 3.Top-down 4.Sandwich

How do you carry out boundary value testing?

1.Identify the ECs. 2.Identify the boundaries of each EC. 3.Create test cases for each boundary value by choosing one point on the boundary, one point just below the boundary, and one point just above the boundary.

What is function testing and what guidelines are there?

A function test (you test one function at a time) checks that the integrated system performs its functions as specified in the functional requirements. The guidelines are: -Use a test team independent of the designers and programmers -Know the expected actions and output -Test both valid and invalid input -Never modify the system just to make testing easier -Have stopping criteria

What is a merge request and what are the pros and cons of using it?

A merge request is created when you submit your code for consideration for inclusion in the main repository. The pros and cons are: +An opportunity to review code and make changes +Less risk of needing rollback -Put extra work on the repository owner -Can increase time before changes are integrated

What is a pilot test?

A pilot test installs the system on an experimental basis. Users exercise the system as if it had been installed permanently. Whereas benchmark tests include a set of special test cases that users apply, pilot tests rely on the everyday working of the system to test all functions. By definition, a pilot test is much less formal and structured than a benchmark test.

Which are the different testing levels and what do they test?

Acceptance test (release testing) - validates requirements and specification System testing (integration testing of modules) - verifies system design (architecture, high-level design) Module testing (integration testing of units) - verifies module design (program design, detailed design) Unit testing - verifies implementation of units (classes, procedures, functions)

What is bottom-up testing?

Bottom-up testing is when each component at the lowest level of the system hierarchy first is tested individually with the help of drivers. Then, the next components that are tested are the one on the next level in the hierarchy, which are calling on the components which were tested first. This approach is followed repeatedly until all components on all levels are included in the testing.

What is boundary value testing?

Boundary value testing focuses on the boundaries simply because that is where so many defects hide. The defects can be in the requirements or in the code.

What are the applicabilities and imitations of testing with equivalence classes?

Boundary value testing is equally applicable at the unit, integration, system, and acceptance test levels. All it requires are inputs that can be partitioned and boundaries that can be identified based on the system's requirements. It makes the assumption that the implementation is consistent for entries of the same type.

Describe equivalence class testing.

Equivalence class testing is a type of black box testing technique used to reduced the number of test cases to a more manageable level, in comparison to exhaustive testing, while still maintaining reasonable test coverage. Each equivalence class (EC) consists of a set of data that is treated the same by the module or that should produce the same result. Any data value within a class is equivalent, in terms of testing, to any other value.

What are the applicabilities and imitations of testing with equivalence classes?

Equivalence class testing is most suited to systems in which much of the input data takes on values within ranges or within sets. It also makes the assumption that data in the same EC is, in fact, processed in the same way by the system. The simplest way to validate this assumption is to ask the programmer about their implementation. EC testing is equally applicable at the unit, integration, system, and acceptance test levels. All it requires are inputs or outputs that can be partitioned based on the system's requirements.

Describe exhaustive testing.

Exhaustive testing is a type of black box testing where every member of the input value space is tested. The input value space is the set of all possible input values to the program.

How should the testing be done in terms of planning?

First of all, there should be a plan for the testing. There is no point in testing without one. This plan defines at what stages that certain types of testing will be performed.

What is a benchmark test?

In a benchmark test, the customer prepares a set of test cases that represent typical conditions under which the customer prepares a set of test cases that represent typical conditions under which the system will operate when actually installed. The customer evaluates the system's performance for each test case.

What is big-bang testing and when can it be used?

In big-bang testing, you first perform unit tests for all isolated units and then test all units together at once. Big-bang testing can be used for small systems but it is not practical for large ones. You don't need drivers or stubs but finding the cause of any failure.

What is a parallel test?

In parallel testing, the new system operates in parallel with the previous version. The users gradually become accustomed to the new system but continue to use the old one to duplicate the new. This gradual transition allows users to compare and contrast the new system with the old. It also allows skeptical users to build their confidence in the new system by comparing the results obtained with both and verifying that the new system is just as effective and efficient as the old one.

What is integration testing?

Integration (and system) testing involves testing the integrated functionality of the complete system.

What is software reliability and how is it measured?

Software reliability is the probability that a system will operate without failure under given conditions for a given time interval. We express reliability on a scale from 0 to 1. A system that is highly reliable will have a reliability measure close to 1 and an unreliable system will have a measure close to 0. Reliability is measured over execution time, not real time (i.e., not clock time) in order to more accurately reflect system usage.

What is software testing?

Software testing is an activity in which a program is executed under specified conditions, the results are observed, and an evaluation is made of the program.

Who does the testing?

The testing can either be done by an independent tester or by the developer. If it is an independent tester, he must learn about the system which will be a bit time-consuming. However, he will attempt to break it "in the name of quality", which will improve the overall quality. If the developer tests the system, he won't hav to spend time on understanding it since he has built it. However, he will test it "gently" and is driven by "delivery" rather than "quality" which the independent tester is.

What is the objective of unit testing?

To ensure that the code has implemented the design properly.

What is top-down testing?

Top-down testing is when the top level, usually one controlling component, is tested by itself. Then, all components called by the tested component(s) are combined and tested as a larger unit. This approach is reapplied until all components are incorporated.

What is validation and what is verification?

Validation: Are we building the right system? Verification: Are we building the system right?

What is xUnit?

xUnit is a set of tools for regression testing where x denotes a programming language. JUnit for Java is one of the earliest and most popular.

Describe GUI testing.

-GUI applications are event driven and users can cause any of several events in any order. This means that there are an infinite number of testing sequences. -Unit testing is typically at the "button level", meaning that buttons have functions and these can be tested in the usual unit-level sense. -The essence of system-level testing for GUI applications is to exercise the event-driven nature of the application -GUI testing is more expensive/harder to automate. However, consistent unit testing at lower levels can help reduce the cost. -The challenges of GUI testing is repeatability and regression

When would it be beneficial to use bottom-up testing?

-If the basic functions are complicated, error-prone or has development risks -If bottom-up development strategy is used -If there are strict performance or real-time requirements

What are the problems with bottom-up testing?

-Lower level functions are often off-the-shelf or trivial -Complicated user interface testing is postponed -End-user feedback is postponed -You need to use drivers (which will take some effort to write)

What are the benefits of TDD?

-TDD makes your work aligned with the requirements as you first have to state the requirements as a test case, and then implement them. -It helps guide you to only write small pieces of code without testing. -It is an effective technique that helps programmers quickly build reliable code, however, it is not a substitute for other quality control activities. -TDD usually leads to writing more tests, and simpler code. In addition to the above benefits, it is important to note that test-driven development should be combined with formal unit and integration testing, code inspections, and other quality control techniques to ensure a high-quality implementation of the requirements. Finally, test cases in TDD are produced based on the developer's intuitions and experience, although other techniques may be used.

What are the problems with top-down testing?

-Technical details are postponed, potential show-stoppers -Many stubs are required -Stubs with many conditions are hard to write

What are the benefits of top-down testing?

-Test cases are defined for functional requirements of the system -Defects in general design can be found early -Works well with many incremental development methods -No need for drivers

What are the guidelines for identifying the equivalence classes?

1.If an input condition specifies a range of values. identify one valid EC and two invalid EC. 2.If an input condition specifies the number (e.g., one through six owners can be listed for the automobile), identify one valid EC and two invalid EC (e.g., no owners, more than six owners) 3.If an input condition specifies a set of input values and there is reason to believe that each is handled differently by the program, identify a valid EC for each and one invalid EC. 4.If an input condition specifies a "must be" situation (e.g., first character of the identifier must be a letter), identify one valid EC (it is a letter) and one invalid EC (it is not a letter) 5.If there is any reason to believe that elementas in an EC are not handled in an identical manner by the program, split the equivalence class into smaller equivalence classes.

Which are the 10 different severity classes for faults? Give examples of a fault for each severity class.

1.Mild: misspelled word 2.Moderate: misleading or redundant information 3.Annoying: truncated names, bill for $0.00 4.Disturbing: some transaction(s) not processed 5.Serious: lose a transaction 6.Very serious: incorrect transaction execution 7.Extreme: frequent "very serious" 8.Intolerable: database corruption 9.Catastrophic: system shutdown 10.Infectious: shutdown that spreads to others

Why are stubs used in top-down testing?

A component being tested may call another component that has not yet been tested, so we write a stub which simulates the activity of the missing component. The stub then answers the calling sequence and passes back output data that lets the testing process continue.

What is a driver?

A driver is a pretend module that requires a sub-system and passes a test case to it. Drivers are used in bottom-up testing

What is a failure?

A failure (anomaly) occurs when a fault executes. Two subtleties arise here: one is that failure only occur in an executable representation, which is usually taken to be source code, or more precisely, loaded object; the second subtlety is that this definition relates failures only to faults of commission. How can we deal with failures that correspond to faults of omission?

What is the definition of a fault?

A fault is the result of an error. It is more precise to say that a fault is the representation of an error, where representation is the mode of expression, such as narrative text, data flow diagrams, hierarchy charts, source code, etc. Defect is a good synonym for fault, as is bug. Faults can be elusive. When a designer makes an error of omission, the resulting fault is that something is missing that should be present in the representation.

What is a fault of commission and omission?

A fault of commission occurs when we fail to enter something into a representation that is incorrect. Faults of omission occur when we fail to enter correct information. Of these two types, faults of omission are more difficult to detect and resolve.

What is a stub?

A stub s a program or a method that simulates the input-output functionality of a missing sub-system by answering to the decomposition sequence of the calling sub-system and returning back simulated or "canned" data.

What is an oracle?

A test oracle is a hypothetical person or machine that can tell whether the actual test results are the same as the expected results. There are two types of oracles: -Human: an expert that can examine an input and it's associated output and determine whether the program delivered the correct output for this particular input. -Automated: a system capable of performing the above task.

What is black box testing?

A testing methodology where test cases are derived from the requirements without consideration of the actual code content.

Which different types of faults are there?

Algorithmic: division by zero Computational and precision: order of operations Documentation: when the documentation does not match what the program actually does Stress and overload: when the data structures (e.g. dimensions of tables, size of buffers, etc.) are filled past their specified capacity Capacity and boundary: when the system activity reaches its specified limits (e.g. x devices, y parallell tasks, and z interrupts) Timing and coordination: when the code coordinating real-time events is inadequate (happens to real-time systems) Throughout and performance: when the system does not perform at the speed prescribed by the requirements Recovery: when a failure is encountered and the system does not behave as the designers desire or as the customer requires. Hardware and system software: can arise when the supplied hardware and system software do not actually work according to the documented operating conditions and procedures Standards and procedures: may not always affect the running of the programs, but they may foster an environments where faults are created as the system is tested and modified. By failing to follow the required standards, one programmer may make it difficult for another to understand the code's logic or to find the data descriptions needed for solving a problem.

What is the definition of an error?

An error is something that a person make. A good synonym is mistake. When people make mistakes while coding, we call these mistakes bugs. Errors tend to propagate; a requirements error may be magnified during design and amplified even more during coding.

What is configuration management?

Configuration management involves maintaining a correspondence among the requirements, the design, the implementation, and the tests. This cross-reference tells developers what program to alter if a change in requirements is needed, or what parts of a program will be affected in an alteration of some kind is proposed. Configuration management staff also coordinate the different versions of a system that may be built and supported. For example, a software system may be hosted on different platforms or may be delivered in a series of releases. Configuration management ensures that the functionality is consistent from one platform to another, and that it doesn't degrade with a new release.

What are the pros and cons of sandwich testing?

Pros: -Top- and bottom-layer tests can be done in parallel -It allows integration testing to begin early in the testing process -It combines the advantages of top-down and bottom-up testing control and utilities from the very beginning Cons: -Does not test the individual sub-systems on the target layer thoroughly before integration

What is sandwich testing?

Sandwich testing combines a top-down strategy with a bottom-up one. The system is viewed as three layers, just like a sandwich: the target layer in the middle, the levels above the target, and the levels below the target. A top-down approach is used in the top layer and bottom-up on is used in the lower layer. Testing then converges on the target layer. The target layer is chosen on the basis of the system characteristics and the structure of the component hierarchy. For example, if the bottom layer contains many general-purpose utility programs, the target layer may be the one above, in which lie most of the components using the utilities.

What is test-driven development and which steps are included?

Test driven development is when you write the unit tests before writing the code, and then using them to make sure that the code works. TDD assumes that you proceed in the following fashion: 1.Write the test case 2.Verify that the test case fails 3.Modify the code so that the case succeeds. (The goal is to write the simplest code you can, without worrying about future tests or requirements). 4.Run the test case to verify that it now works and run the previous test cases to verify that the code has not regressed with any newly introduced defect. 5.Refactor the code to make it pretty.

What is installation testing?

The final round of testing involves installing the system at user sites. If acceptance testing has been performed on-site, installation testing may not be needed. However, if the acceptance testing conditions were not the same as the actual site conditions, additional testing is necessary. Regression tests may be administered to verify that the system has been installed properly and works "in the field" as it did when tested previously. When teh customer is satisfied with the results, testing is complete and the system is formally delivered.

What different kinds of performance tests are there?

The performance test tests your system in accordance to the specified non-functional requirements. Some examples of performance tests are: Stress test Timing test Volume test Configuration test Compatibility tests Regression tests Security tests (Physical) environment test Quality test Recovery test Maintenance test Documentation tests Human factor tests / usability tests It all depends on what your non-functional requirements are.