A/B Testing

What is the goal of A/B Testing?

The goal of A/B Testing is to identify any changes to the web page or product to maximize or increase the outcome of an interest. An example for this could be identifying the click through rate for a banner ad.

Conversion Rate

number of visitors that completed goal/ total number of visitors

Why does A/B testing run both A and B conditions simultaneously

so both conditions are exposed to similar seasonal variables.

Hypothesis of an A/B test is

what is the possible outcome of the experiment

Describe the process of A/B testing

1. Think of new ideas 2. Run experiments 3. Statistically analyze results 4. Update Winning Idea - Repeat

Before running power analysis you should know 1. 2. 3. 4. 5. 6.

1. statistical test you plan to run 2. baseline value: the value of the control condition 3. desired value: the value of the test condition 4. the proportion of data from the test condition (ideally half 0.5) 5. the significance threshold where effect is significant (0.05) 6. the power: 0.8

An average conversion rate are about ________________

20 percent

SSizeLogisticBin is for

Calculating sample size for simple logistic regression with binary predictor

Power analysis

a statistical method to determine the acceptable sample size that you need to ensure an effect is real. - determines how long to run an experiment.