BUS 421 ch. 8
_____ = A/B test that tests more than two conditions
A/B/N test
____ = a controlled experiment using a computer algorithm that tests 2 conditions.
AB testing
What is true about a *bandit test*? - It's an A/B test but w an extra exploitation stage - It shifts traffic in reaction to real-time performance - Its an A/B test with a hidden exploration stage - It removes people from the exploitation stage
It shifts traffic in reaction to real-time performance
_____ = measure of whether a research finding is meaningful because it is unlikely the finding has occurred by chance or error
Statistically significant
The inputs and outcomes that can be tested in A/B testing are *limitless*. T or F
T
T or F: In order to understand why, be sure that A vs. B is EXACTLY the same content EXCEPT for the "why" you wish to test.
TRUE
Crowdsourcing is the practice of gathering information by requesting the services of a large number of people, typically through online connection. For example, we could pay 10,000 people to respond to an A/B Test of our website. What is the downside to this?
The downside is that directing crowdsourcing communities to an A/B test makes is *less natural* and it is not clear whether these people's behaviors would represent real customers well.
A/B Tests are rooted in ____ design
experimental
*** Bandit testing = A/B test methods with an adaptive (exploration / exploitation) stage
exploitation
A false positive occurs when a condition seems to be the winner in the (exploitation/ exploration stage), but it doesn't perform as well in the (exploration/exploitation stage.)
exploration exploitation
A/B tests are more similar to (field/lab experiments)
field
A and B should have (different/identical) content except for the hypothesized elements
identical
An "Experimental condition" is the state of the (independent / dependent) variable for which the (independent/dependent variable) is measured in order to perform statistical calculations
independent dependent
It's called A/B testing because
oftentimes 2 conditions are tested and these conditions are called condition A and condition B
______ = practice of using chance methods to assign participants to experimental conditions
randomization
In the exploitation stage of A/B testing, the successful campaign would go to all of the remaining people on the list, but *bandit testing* would
send both the successful & unsuccessful campaigns to another smaller portion in a series of exploration stages, increasingly using the more successful campaign. The software would continue to track performance iteratively with further adjustments over time.
****____ = probability that a statistical test will reject the null hypothesis given that it is true
significance level
Exploration Stage =
stage of an A/B test for determining which version is more successful with a smaller portion of the potential audience
Exploitation Stage =
stage of an A/B test that applies the findings to a larger portion of the potential audience
******____ = probability that a statistical test will reject the null hypothesis given that is false
statistical power
The chances of a false positive are much *higher* when (too many/ too few) dimensions are tested at once, especially without ample sample size.
too many
**** ____ = An end-point in time for having the data collected
Fixed Horizon means:
Goal of A/B testing is to:
Identify differences in marketing outcomes after random assigning of people to marketing input A or marketing input B
Other names for A/B testing are (2)
Split testing (bc splitting our audience) Bucket testing (placing audience members in diff buckets to see response to diff marketing inputs)
Key issues w A/B testing:
- ignoring small gains - incomplete timing - It is good to get a full week, so end the exploration stage at the same exact day and time it was started - multiple concurrent tests - underestimating false positives - poorly designed hypotheses -- we recommend running A/B tests with well designed hypotheses that consider more than whether A is more successful than B, but rather *why it is more successful*
**** Steps of scientific method:
1. Ask a question tied to a specific decision or action that needs to be made 2. Formulate hypotheses with testable predictions to answer the question 3. Collect data to test the predictions 4. Analyze the data 5. Accept, change, or reject the hypotheses & repeat until the question has been answered
A/B testing occurs in how many stages?
2
The diff b/w A/B Tests & Experiments is that
A/B Tests are often about ENDS rather than MEANS Experiments test hypotheses, seek to understand outcomes and reasons, WHY This is why you should approach A/B tests as Experiments, to understand WHY!!!
_____ = results of the *exploration stage* of the A/B test lead to the belief that one of the options A or B is more successful than the other, when in reality they are likely to perform the same in the exploitation stage
False positive
AB testing is a type of (controlled or uncontrolled) experiment?
Controlled
**** Type of experiment in which a hypothesis is tested by looking for changes in a dependent variable measure caused by manipulated changes to an independent variable as the *only* factor that is allowed to be adjusted
Controlled experiment
What are the 2 stages of A/B testing?
Exploration Exploitation
______ = results of the *exploration stage* of the A/B test lead to the belief that options A or B do not differ in their success, when in reality one is likely to outperform the other in the exploitation stage
False negative
A/B testing tools:
Optimizely Google Experiments - A/B testing for GA Ab Tasty Qubit Adobe Target SiteSpect Convert Experiences VWO Sentient Ascend
****_____ = procedure consisting of systematic observations for testing hypotheses
Scientific Method
_____ = incorrect rejection of a true null hypothesis (a "false positive")
Type I error null hyp =the hypothesis that there is no significant difference
____ : incorrect rejection of a false null hypothesis (a "false negative")
Type II error