CSE 201 Test 4

Book definition of Black Box

"a system or component whose inputs, outputs, and general function are known but whose contents or implementation are unknown or irrelevant" "test cases rely only on the input/output behavior"

Book definition of Glass (White) Box

"a system or component whose internal contents or implementation are known" "tests rely on information about how the software has been designed or coded" Control-flow-based testing

Book definition of Regression Testing

"retesting to detect faults introduced by modification" "when you fix one bug, you introduce several newer bugs"

Book definition of Unit Test

"testing of individual routines and modules by the developer or an independent tester"

Who should system test?

(1) Try to have a different set of people doing your system testing. (2) Never system test your own code. You know it too well to be unbiased.

What are the 3 things iterations help you with?

(1) Your team stay focused (2) Deal with just a manageable amount of user stories (3) Keep you from getting too far ahead without built-in checkpoints with your customer.

Continuous Integration Cycle (reference diagram in Lecture17 page 11)

(Source Control) > Initiate CI Process > (Build) > Test > (Testing) > Report > (Development) > Commit > repeat

Life & Death of a Bug (6)

- A test finds a bug - The tester files a bug report - create a story (or task) to fix the bug - fix the bug - check the fix - update the bug report

Example of Cruise Control

- Add your JUnit test suite to your Ant Build - Create CruiseControl Project - Check repo and get latest code - Schedule the build

Working software means...

- Completing your iteration's work - Passing all your tests - Satisfying your customer

Cruise Control

- Continuous Integration Tool - Also an extensible framework for creating custom continuous build processes

When the iteration is almost complete what list of things should you go through...

- Customer-driven functionality - Compiling code - Monitored builds - Continuously tested code - Solid test coverage - Reliable progress tracking - Pacing that adapts to the team

What is the purpose of testing?

- Examine design and code - Improve the quality of verification, validation, confidence, and detecting failures

Calculating the velocity allows you to ...

- Expect what you can get done in the NEXT iteration - Shows you the REALITY of how you and your team develop - How you can plan your next iteration for success

What are the 3 purposes of an abuse case?

- Identify potential vulnerabilities - Elicit security requirements - Created together with a use case diagram

4 Steps in the Abuse Case process?

- Identify the actors - Identify the abuse cases - Define abuse cases - Check granularity

What are 3 testing frameworks?

- JUnit - Check - Unity

What do the testers actually look at during grey box testing?

- Looking at the data is your database to make sure things are being cleaned up correctly - Checks that ports are closed - Network connects dropped - And that memory usage is staying steady

What are elements you should include in your iteration review?

- Prepare ahead of time and think introspectively (evaluate yourself) - Be forward - looking/constructive, it's ok if things aren't perfect but don't turn iteration reviews into whining sessions - Calculate your metrics to convey if the velocity went up or down - Have a standard set of questions to review, have recurring questions.

When you've completed your products, what list of stuff is could always be done...

- Process improvements - System testing - Refactoring of code using lessons you've learned - Code cleanup and documentation updates - More design patterns - Development environment updates - Research & Development on a new technology you're considering - Personal development time to let people explore new tools or read

2 types of bugs

- Programming errors - Functional issues that no one picked up on when writing their user stories

What are other sources of defects?

- Programming logic/coding error - Testing error - Bug fix that introduces another error

What should you do to adjust for every iteration?

- Recalculate estimates & velocity at each iteration - Applying the things you learned from the previous iteration

Track bugs and...

- Record and communicate priorities - Keep track of everything including discussion, code changes, and decisions - Generate metrics including bug submission rate, where they come from, and priority

When building your suite, testing should be considered at every phase, the 3 things that should be included are..

- Requirements (verification) - Design - Construction

What are two things you should do to involve the customer?

- Talk to them when things change - If the impact is significant, in terms of functionality or schedule, then you've got to go back to the customer

Your system test must:

- Test a system that actually works - Have a dedicated end-to-end period or let the real users work on the system (even if it's on a beta release) - Test the system in a situation that's as close to real-world as you can manage.

Why should testing always be done by testers?

- Tester can often bring a fresh perspective to the project by approaching the system with a fundamentally different view - They're looking for bugs, so they don't how sleek your code is, they just want the system to work.

What are the 5 General Principles?

- Testing shows the presence of defects, not their absence - Exhaustive testing is not possible - Testing activities should start as soon as possible - Defects tend to cluster together - Tests are dependent on use and environment.

Why should we Automate?

- To verify & validate the requirements and organization - To find defects & failures caused by misunderstood requirements or coding errors - compatibility of the software

Continuous Integration

- When a task is complete it is integrated into the whole system - After any such integration, all the unit tests in the system must pass

How can untested code create security issues?

- You need to be careful that code that isn't tested isn't available for people to use - either accidentally or deliberately - Your coverage report can help identify which code you're actually using

3 things that make up a test suite

- library of unit tests - one command - all tests run at once

2 most important things you should do when dealing with bugs....

- record them - track them

While working through an iteration, be sure to dedicate time to...

-Bug fixes - Take this off of the available hours when you do your iteration planning, so you don't need to worry about it affecting your velocity

What is the Mark II?

1947 Harvard computer that took up an entire room. Includes mechanical relays, glowing vacuum tubes, and was worked on by technicians who would program the computer by reconfiguring it and change the occasional vacuum tube.

How many iteration cycles does system testing require?

2

What was the first computer bug?

A moth flew into the computer and was zapped by the high voltage when it landed on a relay.

What's the difference between a Use Case and an Abuse Case?

A use case is a complete transaction between one or more actors and a system. An abuse case is the exact same except it results in harm.

Should code be perfect or just good enough?

As long as any problems in the code don't result in bugs (or software bloat), and you deliver the functionality that your customer needs, then you'll be successful

Integration Testing

Assemble groups of modules,classes,etc and test collaboration

Where do unit tests work best?

Automated test suites but can potentially miss bugs that only show up when components interact, or when real, live users start banging on your system.

In system testing, you should treat the system like a ____ _____.

Black Box

5 Types of Testing

Black Box, Grey box, White Box (Glass Box), Unit Testing, Regression Testing

The bottomline: all software has some bugs

By applying your process you can avoid those bugs rearing their ugly head in your software's functionality.

3 Steps of Spike Testing

Code for a little > See what you get done > Estimate how long everything else will take

What's the key to most problems you'll run into during software development?

Communication. When in doubt, talk to your team, other teams, and your customer.

What are Abuse Cases?

Complete interaction between a system and one or more actors, where the results of the interaction are harmful to the system, one of the actors, or one of the stakeholders in the system.

If you find a bug while you're working through iterations, you should...

Determine if it's important enough to fix now, or put off til later. Treat them like stories: prioritize and bump lower-priorities

White Box View (Glass Box)

Developers are deep in the trenches. They see: - design - patterns - duplicated code - AND inconsistencies in how things are represented

2 free & 2 commercial bug tracking softwares

Free: Bugzilla and Mantis Commercial: TestTrackPro and ClearQuest

System testing mainly exercises the _______ of a system.

Functionality

Test Coverage Tools

Generate a report telling you how much of your code is covered

What should be the first priority?

Getting things buildable! At least you should have a little bit of control over the code.

What do the results from a spike test tell us?

Gives an idea of how long it will take to fix the remaining bugs

What is included in the test case of a Black Box Test?

Inputs + pre-test state of the software and the expected results will be outputs of a state.

What does Fixing Functionality mean?

Only fixing the bugs that directly affect functionality - focus just on the functionality in you user stories

Equivalence Partitioning helps...

Partition the test cases into "equivalence classes" and each equivalence class contains a set of "equivalent" test cases

How do defects arise from requirements or design activities?

Requirements get misinterpreted or recorded incorrectly. Incorrect design specification or specification is incorrectly interpreted.

What's one downside of the White Box View?

Sometimes they see so much detail, it's possible for them to miss broken functionality or make an assumption that a tester or end user might not.

How do we estimate fix time?

Spike Testing - work on the code for a little > see what we can get done > extrapolate out from that

Grey Box View

Testers peek under the covers a little. Looking for functionality but they're usually poking underneath to make sure things are really happening the way you said they would.

Difference between Testing and Debugging

Testing finds the bugs & Debugging finds the cause and fixes the bugs

What do spike tests tell us?

Tests gives you an idea as to how much of your code was failing

How can you determine what functionality is working?

Tests! Create unit tests for any new integrated parts! If you can't test it, it's broken!

Build Automation

The process of converting source code files into standalone software artifacts that can be run on a computer, or the result of doing so.

Why aren't unit tests enough?

Unit tests only focus on small pieces of code, one at a time, and deliberately try to isolate components from each other to minimize dependencies.

Black Box View

Users see the system from the outside. It either does what they ask it to do or it doesn't. They only care about functionality.

Why is Velocity so important?

Velocity = RealWorld. Velocity is a key part of planning. It's looking for a value that corresponds to how fast your team actually works, in reality, based on how they've worked in the past.

Continuous Integration (CI) Formula

Version Control + Compilation + Testing = CI Automated Process

Using Spike Tests only gives you an estimate because

When it comes to bug fixing, you really can't be sure. Spike tests give you quantitative data upon which you base your estimates.

Is it okay to only test functionality?

Yes, there are probably large pieces of your library that haven't yet been covered by tests but you have tested all the code that you actually use to complete your user stories.

How does performing spike tests on a random sample of bugs help you estimate how long it will take to fix the total number of bugs?

You know how many bugs you fixed, and it was a random sample, so you can say with a certain degree of confidence that you should be able to fix the same number of further bugs in roughly the same amount of time.

Fix Bug Rate Formula

You should calculate this at the end of your spike testing: (Total bugs your entire team fixed) / (Number of days in the spike test) = Your daily bug fix rate

Why is it not practical to always hit 100% coverage? What amount of coverage should you shoot for?

You'll get diminishing returns on your testing after a certain point. Aim for about 85% - 95% coverage because it's just not possible to tease out that last 10% - 15%.

3 ways of looking at a system

black box, white box, grey box

No matter how careful you are there will always be ______.

bugs

Don't forget to get approval from the ____ on the _____ _____.

customer on the overall project

System Testing is about...

reality, and the system as a whole, rather than all its individual parts