QA Testing (graded)

- What was the response time on average for the "calculator" test?

32.32s

- What was the response time on average for the "google speed test" test?

83.354s

A response to an HTTP request than can be stored by the client and reused later without repeating the request again is called ...

Cacheable

- First, think about what you tested in this JMeter exercise. What does the response time of each test mean to us?

GOOD 5m38s

"Simple checks on programs for naming conventions are not useful for detecting or preventing defects". Provide one argument or reason to counter this statement.

Good

What type of object does the example use to collect performance samples?

SampleResult

- What factors do you think influenced the response time of your tests?

internet speed

- How could you make our performance test of the google speed test more robust such that it recognizes failures of the software under test faster?

performance test of the google speed test can be made more robust by testing the application output, processing speed, memory utilization. Command response time and network bandwidth

- Study the (above/below) graph of the "google speed test" test running on a machine. Why do you think some response times on the graph are drastically different?

response times on the graph are drastically different because of network speed variation as per ISP or slow process of individual's computer

A safe HTTP method will ...: • ... not get logged on the server • ... make the webserver enforce read-only semantics for the application • ... also be idempotent • ... not lead to a change of state of the resource on server

• ... also be idempotent • ... not lead to a change of state of the resource on server

What is the general order of steps for interacting with a web app? • 1. Start Browser 2. Open Webpage 3. Locate Element 4. Examine Output 5. Send Input • 1. Start Browser 2. Open Webpage 3. Locate Element 4. Send Input 5. Examine Output • 1. Start Browser 2. Open Webpage 3. Send Input 4. Locate Element 5. Examine Output • 1. Start Browser 2. Open Webpage 3. Examine Output 4. Locate Element 5. Send Input

• 1. Start Browser 2. Open Webpage 3. Locate Element 4. Send Input 5. Examine Output

Successful automated testing always requires • A machine-checkable oracle that can determine success or failure. • Access to the internals of the program to monitor program state. • Access to really fast computers to generate the tests. • All of the above

• A machine-checkable oracle that can determine success or failure.

What are some things an attacker may do when injecting SQL code into a database? • Add data • Delete data • Collect data • Modify data

• Add data • Delete data • Collect data • Modify data

What can cause difficulty in adequately testing web and mobile applications? • Poor internet connection • Screen size • Differences in OS • All of the above

• All of the above

What steps can be used to improve fuzzing performance? • Biasing inputs for random testing • Running multiple fuzzers concurrently • Running each fuzzer for longer periods of time • Adding grammar information to the fuzzer • All of the above

• All of the above

Which of the following describes the problem with testing machine learning systems? • No traditional program structure for test metrics • Difficult to define requirements for program behavior • Online evolution means that program behavior will change • All of the above

• All of the above

Why is it important to think about risks when testing your application's security? • Certain attacks can be more harmful to the budget of the business. • Nonmalicious users could be harmed. • Attackers can remain in a system undetected. • All of the above

• All of the above

Concurrent systems are quite difficult to test because: • the different potential interleavings of threads leads to 'race conditions' where the program may behave differently between executions. • to ensure that only one thread uses a variable at a time, Java 'synchronized' code can lead to deadlocks where each thread is blocked waiting for another thread. • All of the above. • the different potential interleavings of threads means that there are many more possible system states.

• All of the above.

Parents are a part of what axes? • Ancestors • Descendents • Preceding • Following

• Ancestors

Executing a program P on three different inputs 1, 10, and 100, we observe the respective outputs to be 0, 1 and 2. Which of the following can be inferred from this observation. • Any correct description of the behavior of the program should allow the pairs (1, 0), (10, 1) and (100, 2) for the value of (input, output). • The program outputs the number of digits of the input that are zero. • The property "output is a non-zero integer" is not satisfied by this program. • Suppose we conclude from the three observations that the property "output is less than input" holds true for this program. This could be a potential case of optimistic inaccuracy.

• Any correct description of the behavior of the program should allow the pairs (1, 0), (10, 1) and (100, 2) for the value of (input, output). • The property "output is a non-zero integer" is not satisfied by this program. • Suppose we conclude from the three observations that the property "output is less than input" holds true for this program. This could be a potential case of optimistic inaccuracy.

Program annotations are useful for providing information to tools that analyze the programs. Such annotations • Carry metainformation, i.e., provide information about program elements • Can be used only for static analysis, i.e., the information cannot be made available at runtime • May affect program behavior, i.e., the semantics of the program may change

• Carry metainformation, i.e., provide information about program elements

What is the main source of human effort for grammar-based fuzzing? • Writing tests • Manually instrumenting the program • Creating the grammar to be used by the fuzzer • Determining what kind of fault was demonstrated by the test • All of the above require similar levels of human effort

• Creating the grammar to be used by the fuzzer

What type of cross-site scripting attack is likely the most serious? • Stored data • Active data • Reflected data • DOM style attacks

• DOM style attacks

The main problems with symbolic execution are: • Dealing with program paths containing non-linear math • Handling complex string operations • Choosing the correct value that solves complex linear constraints • Too many paths in large programs • Program dependencies (like databases) that are unknown to the symbolic solver

• Dealing with program paths containing non-linear math • Handling complex string operations • Too many paths in large programs • Program dependencies (like databases) that are unknown to the symbolic solver

What is most likely the fastest possible time to release new versions of a website? • Every second • Every minute • Every day • Every week

• Every second

If a request PUT /file.html HTTP/1.1 is made twice it will result in creating two copies of the resource file.html on the sever. • TRUE • FALSE

• FALSE

Automated testing should replace writing test cases by hand. • True • False

• False

Cascading style sheets are made to only modify HTML documents. • True • False

• False

Fuzz security testing is only concerned about generating malformed inputs. • False • True

• False

Fuzz testing is defined as the use of automation to slowly build an understanding of code in order to purposefully break it. • True • False

• False

If the program returns an error due to fuzz testing, there is likely a bug in the program. • True • False

• False

In web development, it is required to fully test all possible browsers and form factors before releasing the product to the customers? • True • False

• False

Insecure deserialization can corrupt data, but it's not actually possible to use it to break into a system. • True • False

• False

It is not possible for performance testing to affect users currently using a service. • True • False

• False

It is virtually impossible for attackers to mimic a legitimate user as there are many factors that reveal their true identity. • True • False

• False

It should be ensured that all websites are fully loaded in the span of 10 seconds. • True • False

• False

Knowing one testing technique is enough to be an effective test engineer. • True • False

• False

METS defines the standard of 50% of testing goes to direct testing, 25% to proximate testing, and 25% to regression testing. • True • False

• False

Metadata is defined as data that produces other data. • True • False

• False

None of the automated verification techniques scale to real programs. • True • False

• False

Numbers are represented the same in all computers to ensure that there are no errors in communication. • True • False

• False

OWASP identifies only the vulnerabilities that developers have come across since its creation in 2003. • True • False

• False

Setting a web element at a specific pixel location is robust and effective. • True • False

• False

Throughput and responsiveness are generally defined as the same thing. • True • False

• False

Web applications never store data locally on the computer. • True • False

• False

'Service' is the system behavior as defined by the software requirements. • True. • False.

• False.

Choose the features of an education website which are most likely to be of high criticality for testing. • Finding related courses for currently enrolled students • Checking whether a course fits into a degree program • Full course catalog is available to all users • Course search functionality

• Full course catalog is available to all users • Course search functionality

Which HTTP request methods are allowed (or typically used) in a HTML Forms? • GET • DELETE • POST • PUT

• GET • POST

How is smart grammar-based fuzz testing different than adaptive random testing? • Grammar-based fuzzing is more likely to generate more almost-correct malformed inputs than adaptive random testing • Grammar-based fuzzing has more knowledge of valid inputs • They are not substantially different • Adaptive random testing uses more sophisticated metrics for generating the next test

• Grammar-based fuzzing is more likely to generate more almost-correct malformed inputs than adaptive random testing • Grammar-based fuzzing has more knowledge of valid inputs

Which of the following statements are not true about HTTP Verbs? • HTTP requests that are successfully handled must be result in a 2XX (e.g. 200) status code in the response. • HTTP requests GET and HEAD are safe . Therefore the application developer need not be concerned about accidental changes to data that affect the server state since the webserver will prevent such changes. • If the HTTP response to the HEAD request includes a body then one should indeed process it like a GET method. • If a HTTP request method is idempotent then the method is safe.

• HTTP requests GET and HEAD are safe . Therefore the application developer need not be concerned about accidental changes to data that affect the server state since the webserver will prevent such changes. • If the HTTP response to the HEAD request includes a body then one should indeed process it like a GET method. • If a HTTP request method is idempotent then the method is safe.

Unit test is used to test: • Design of the software. • All of the above. • Implementation of the software. • User initial requirements.

• Implementation of the software.

The most useful automated verification technique that we have examined is • Random testing • Adaptive random testing • Fuzz testing • Search-based testing • Static analysis • It depends. What is most useful depends on the goal of testing.

• It depends. What is most useful depends on the goal of testing.

Which of the following are true about testing? Check all that apply. • It is difficult to do rigorously. • It can conclusively determine whether the software is correct. • It documents system behavior. • It can sometimes find errors that are not actively looked for, when (for example) a program crashes during execution of a test case. • It checks the whole system, including software that you didn't write.

• It is difficult to do rigorously. • It documents system behavior. • It can sometimes find errors that are not actively looked for, when (for example) a program crashes during execution of a test case. • It checks the whole system, including software that you didn't write.

The main problems with random testing are: • It does not scale to real programs. • It is unable to explore paths that rely on specific inputs given a large input space. • Input distributions in practice may not match random distributions used during testing, so the likelihood of failure may be larger than expected in actual use. • It takes a lot of time to generate each test.

• It is unable to explore paths that rely on specific inputs given a large input space. • Input distributions in practice may not match random distributions used during testing, so the likelihood of failure may be larger than expected in actual use.

What language(s) are used to create software in the Android OS? • JavaScript • Objective-C • Swift • Java

• Java

Which languages are most commonly used with HTML? • Java • JavaScript • XML • CSS (Cascading Style Sheets)

• JavaScript • CSS (Cascading Style Sheets)

Which of the following can be performed with runtime monitoring? • Measuring whether application real-time performance is adequate • Monitoring whether the application meets its functional requirements • Determining whether the program is correct • Measuring whether the environment matches program assumptions • Determining whether the program will terminate

• Measuring whether application real-time performance is adequate • Monitoring whether the application meets its functional requirements • Measuring whether the environment matches program assumptions

What information will you always see about a webpage in Developer Tools? • Current selected element • Network usage • Mouse position • Memory usage

• Network usage • Memory usage

Which are important factors to think about when making a mobile application? • Network usage • Memory usage • Battery usage • Screen usage

• Network usage • Memory usage • Battery usage

Does a program terminating with an error always indicate a failure? • No. • Yes.

• No.

Suppose we have two systems A and B, and A is more rigorously tested than B. Does System A better meet its quality goals than System B? • No. • Yes.

• No.

You are working on a mobile application for Android with a group of team members. One of the team members thinks the team should make the application utilize a large portion of processing power to ensure the app is smooth on all devices. Is this a good idea? • Yes. The user will be happy that there are never any delays while using the software. • No. The user would better appreciate a focus outside of the speed of the application. • Yes. You might as well use as much of the processor as possible if it's available. • No. Some phones may not be able to provide so much of the processor and the phone may suffer elsewhere.

• No. Some phones may not be able to provide so much of the processor and the phone may suffer elsewhere.

What language(s) are used to create software in iOS? • JavaScript • Objective-C • Swift • Java

• Objective-C • Swift

What is a "fail-safe" system? • One in which the software or physical systems can fail, leaving the system in a safe state. • One in which the software makes the system safe in case of physical failures. • One in which there is redundancy so any single physical or computer failure will not cause the system to fail. • All of the above

• One in which the software or physical systems can fail, leaving the system in a safe state.

Are fuzzers optimistic or pessimistic verification tools? • Optimistic - They may miss existing security problems in programs • Pessimistic - They only find crashes or security flaws.

• Optimistic - They may miss existing security problems in programs

The typical HTTP Request that is used to send information to server from a an HTML form is • POST • PUT • PATCH

• POST

Which of the following are true of systems of systems? • Parts evolve independently • No central governing authority • Decoupled execution allows separate testing • Interfaces evolve over time

• Parts evolve independently • No central governing authority • Interfaces evolve over time

A static analysis technique overapproximates the reachable states of a program, to examine those states and determine if the program could ever reach an undesirable state. Such an analysis is: • Pessimistic i.e., the analysis may result in a "yes" answer for some programs do not exhibit undesirable behavior on any input. • Optimistic i.e., the analysis may result in a "no" answer for some programs that exhibit undesirable behavior on some input. • Precise i.e., the analysis result is always correct; the analysis will answer "yes" for exactly those programs that do not reach undesirable states on any input.

• Pessimistic i.e., the analysis may result in a "yes" answer for some programs do not exhibit undesirable behavior on any input.

Assert statements in a program capture expectation of what must be true when execution reaches a certain point in the program. Such expectations may capture: • Post-condition. What is expected to be true after a part of a code is executed. • Pre-condition. What is expected to be true before a part of a code is executed. • Invariant. What is expected to be true every time some code is executed.

• Post-condition. What is expected to be true after a part of a code is executed. • Pre-condition. What is expected to be true before a part of a code is executed. • Invariant. What is expected to be true every time some code is executed.

Siblings are a part of what axes? • Ancestors • Descendents • Preceding • Following

• Preceding • Following

Tasks that can be part of the Tear Down phase are: • Initialize test case values. • Remove data you added after testing is done. • Open connection for testing. • Close connection after testing is done.

• Remove data you added after testing is done. • Close connection after testing is done.

What different performance measures does the example calculate? • Load Testing • Success Recognition • Page Loading • Failure Recognition

• Success Recognition • Page Loading • Failure Recognition

Choose which of the following are true: • Test automation is not recognized as a useful skill in the industry. • Testers will also be expected to develop software in the future. • According to a recent survey, more than 1/2 of businesses are using test-generation tools. • We understand how to test machine learning software like neural nets. • Agile and DevOps are important skills for test engineers.

• Testers will also be expected to develop software in the future. • According to a recent survey, more than 1/2 of businesses are using test-generation tools. • Agile and DevOps are important skills for test engineers.

If a test fails, it could mean: • There is no problem. • The test itself is incorrect. • The program is incorrect.

• The test itself is incorrect. • The program is incorrect.

Why is regression testing well-suited for automated verification? • The tests are low cost to generate so it is inexpensive to throw tests away. • It is straightforward to generate many tests using automated verification. • A good oracle for determining the success of automated tests is always available.

• The tests are low cost to generate so it is inexpensive to throw tests away. • It is straightforward to generate many tests using automated verification. • A good oracle for determining the success of automated tests is always available.

Why is testing considered an 'optimistic' verification technology? • You can use testing on any kind of software, whereas you can only run static analysis or proofs on small systems. • The tests may all pass but the program may still be incorrect. • Any tests that fail may not actually matter from the user's perspective. • Some tests may fail, but the program may be correct.

• The tests may all pass but the program may still be incorrect.

What is the relationship between Java and JavaScript? • They have a similar name. • They are the same language. • They were created by the same people. • They are both used in the same way.

• They have a similar name.

What are Selenium's primary uses? • To give users the ability to interact with the browser. • To act as a user would. • To automate web-browsing. • To create web objects for a webpage.

• To act as a user would. • To automate web-browsing.

What is the goal of fuzz testing? • To check whether software meets its functional requirements • To check whether software has good real-time performance • To check whether software has security bugs • To check whether software is robust (does not crash)

• To check whether software has security bugs • To check whether software is robust (does not crash)

What is the goal of regression testing? • To check whether new functionality works correctly • To determine whether previously developed software performs the same way after modifications • To add to test suites to more rigorously test existing software • To determine whether or not a bug in a previous version of the software is fixed • The first and second answers are true • All of the above

• To determine whether previously developed software performs the same way after modifications

Injection involves tricking an interpreter to run code that it recieved from data submissions. • True • False

• True

It is a simple process to test whether your application can be affected by maliciouos XML entities. • True • False

• True

METS defines a structure that requires testing to be divided between the funtionality of the website as well as the physical objects of the website. • True • False

• True

Making small modifications is important in fuzz testing as this better finds interesting and exploitable results. • True • False

• True

Malicious attackers tend to use similar techniques when breaking into applications. • True • False

• True

Pages are updated automatically as changes are made in Developer Tools. • True • False

• True

Performance is very situational and a complete knowledge of how users percieve performance is always necessary before testing should occur. • True • False

• True

Responsive Design Mode is useful for checking your webpage for different form factors. • True • False

• True

Selenium is used to make your program more robust for browsers. • True • False

• True

The physical testing process focuses on user interactions and the viewing of the webpage while the functional testing process focuses on the creation and animations of the physical objects of the website. • True • False

• True

URLs or Universal Resource Locators act as paths to resources found on machines. • True • False

• True

Universal verbs are used to familiarize all clients and servers with interactions between each other. • True • False

• True

XML and HTML documents can be represented as a tree of nodes. • True • False

• True

XML is designed to describe data while HTML is designed to display data. • True • False

• True

XPaths allow easy navigation into webpages from a language such as Java. • True • False

• True

Did your tests ever have failures? • Yes • No

• Yes • No

If software is not fail-safe, is there any good reason for runtime monitoring? • No - in this case, the system will fail anyway. • Yes - we can give control back to the operator so that they can control a process manually. • Yes - we can provide warnings to the operator that the software may not be behaving correctly. • Yes - this technique lets the software recover from failures and keep going. • No - it slows the program down too much.

• Yes - we can give control back to the operator so that they can control a process manually. • Yes - we can provide warnings to the operator that the software may not be behaving correctly.

Your team's application relies on a lot of other software to run succesfully such as a computer's operating system or a framework to run the code. It has only been a week since the team has checked for updates on all relevant software. Is your application at risk of attack due to this software? • Yes. • No.

• Yes.

The WebDriver object ... • is useful for creating objects on websites. • acts as a framework to interact with the browser driver. • controls the web browser directly. • connects your code to actions on the browser

• acts as a framework to interact with the browser driver. • connects your code to actions on the browser

In the example, performance samples begin... • before an action takes place. • after an action takes place.

• after an action takes place.

Adaptive cruise control software that continues to run in the presence of multiple hardware and service failures but regularly misjudges the distance between cars by a substantial amount is an example of a ______ system. • certainly correct. • certainly robust. • certainly incorrect. • certainly reliable. • certainly safe.

• certainly robust. • certainly incorrect.

Performance testing is a good way to find out ... • common patterns of user activity. • how many users can be actively using a web server. • whether the pieces of a system work well together. • what users expect from your application.

• how many users can be actively using a web server. • whether the pieces of a system work well together.

In the JUnit test framework, we write test cases: • inside the executed method and we annotate that this part is for testing. • inside the class to be tested we annotate that this part is for testing. • in a separate class, and for each method in the program we associated a test case(s) to test the correctness of the method. • All of the above.

• in a separate class, and for each method in the program we associated a test case(s) to test the correctness of the method.

What type of data will the query '//faculty[@name="James Bond"]/../course/@id' from the video return? • <course> • <department> • string • <faculty>

• string

Examples of analyses that can be cast as dataflow analysis problems include: • taint analysis, i.e., is there a potential for tainted input to be used at a particular statement where a sensitive operation is performed? • use-before-initialization, i.e., analysis to determine if there is a potential use of a variable before it is initialized • assertion-violation-detection, i.e., analysis to determine if an assertion violation can occur.

• taint analysis, i.e., is there a potential for tainted input to be used at a particular statement where a sensitive operation is performed? • use-before-initialization, i.e., analysis to determine if there is a potential use of a variable before it is initialized

Direct testing is best defined as • testing that alters the code. • testing that is run after every release. • testing of the feature that has just been added. • testing of the features surrounding a new feature.

• testing of the feature that has just been added.

A latent error becomes an effective error when: • the user types it into the code. • the program starts executing. • the error causes the program to return something unexpected to the user. • the program executes the line of code containing the latent error. • the program reaches a state where the error manifests.

• the program reaches a state where the error manifests.

image 37 The above is a representation of a Java compilation output showing an error in a piece of Java code. In this instance, the error indicated is • the result of a static analysis • the result of executing the code • an indication of a real use-before-initialization fault • an example of a false alarm

• the result of a static analysis • an example of a false alarm

Mutation testing is a _______ metric • gray-box. • black-box. • white-box.

• white-box.