ch 8 bus 421
Which is true about designing A/B tests to extract maximum meaning?
A and B should have identical content except for the hypothesized elements
What is NOT an A/B testing tool
AB Lab
What are some A/B testing tools
AB tasty, Google experiments, Qubit
What is true of a bandit test
It shifts traffic in reaction to real-time preformance
A/B test
controlled experiment that tests two conditions
Fixed Horizon
end-point in time for having the data collected
Type I error
incorrect rejection of a true null hypothesis (a "false positive")
Type II error
incorrect retention of a false null hypothesis (a "false negative")
Promotional campaign
marketing communication designed to present information about, increase demand for or differentiate a product
Statistical significance
measure of whether a research finding is meaningful because it is unlikely the finding has occurred by chance or error
A/B tests are...
more similar to field experiments than lab experiments
Sample size
number of participants to include in an A/B test or other experiment
Digital Marketing Platform
online places for the marketing of good or services
What is NOT a key issue of A/B testing?
overestimating false negatives
Crowdsourcing
practice of gathering information by soliciting the services of a large number of people, either paid or unpaid, typically through online connections
Randomization
practice of using chance methods (e.g, flipping a coin, pulling numbers out of a hat, generating numbers by chance from a computer function, etc.) to assign participants to experimental conditions
Significance level
probability that a statistical test will reject the null hypothesis given that it is true
Scientific method
procedure consisting of systematic observations for testing hypotheses
Hypothesis
proposed explanation of some phenomenon, used as a starting place for further investigation
Availability Bias
reliance on examples that immediately come to mind when evaluating a topic, typically from people's own personal lives
False Positive
results of the exploration stage of the A/B test lead to the belief that one of the options A or B is more successful than the other, when in reality they are likely to perform the same in the exploitation stage
False negative
results of the exploration stage of the A/B test lead to the belief that options A or B do not differ in their success, when in reality one is likely to outperform the other in the exploitation stage
Exploration stage
stage of an A/B test for determining which version is more successful with a smaller portion of the potential audience
Exploitation stage
stage of an A/B test that applies the findings of the exploration stage to a larger portion of the potential audience
Experimental condition
state of the independent variable for which the dependent variable is measured in order to perform statistical calculations
Overconfidence
tendency to think too highly of one's expertise
Statistical power
the probability that a statistical test will REJECT the null hypothesis given that it is FALSE
Controlled experiment
type of experiment in which a hypothesis is tested by looking for changes in a dependent variable measure caused by manipulated changes to an independent variable as the only factor that is allowed to be adjusted
What is a false positive
Incorrect REJECTION of a true null hypothesis
Bandit Testing
A/B test methods with an adaptive exploitation stage
A/B/N test
A/B test that tests more than two conditions
Split testing
A/B testing
Which A/B testing stage typically uses a larger portion of the potential audience
Exploitation phase
What is NOT a type of an A/B test?
Optimize test
What is NOT a crowdsourcing platform
Qualtics
What are some key issues of A/B testing
Underestimating false positives, multiple concurrent tests, ignoring small gains, incomplete timing, and poorly designed hypotheses
What are crowdsourcing platforms
Usability hub, Usertesting, Mechanical turk
Marketing intuition
ability to understand or predict a marketing phenomenon immediately, without the need for data analysis