Course 5 Sec. 7: Testing and Iterating

What is the purpose of an A/B test?

A/B tests are used to test two different versions of something to see which one performs better. A/B tests provide you with data so you know which version performs best and can improve your campaign.

Question: What happens in the iteration stage of the experimentation process?

Answer: During the iteration phase, the test is run again to see if the results are the same. If the results prove to be positive again, the change can be incorporated into the campaign. Then, the process begins all over again; a marketer asks what else could be changed that might improve the campaign.

Question: What stage in the experimentation process is this an example of? "If I change the text on the button to a more engaging call to action, the number of potential customers who sign up for a free trail will increase." 1. Determine what needs to improve. 2. Hypothesize change. 3. Identify variables and create variation. 4. Run experiment. 5. Measure results. 6. Iterate.

Answer: This is an example of hypothesizing change.

What is the key to improving your marketing strategy, campaigns, and ads?

testing and iterating

How to conduct A/B Testing?

Use an A/B Testing tool

What is an example of A/B testing?

A/B testing could compare whether you get more sign-ups on your site with an ad or without an ad.

Define the "control" in an experiment:

In an experiment this is the normal, unchanged version of the element you're testing

Define the "treatment" in an experiment:

In an experiment, this is the new variation you would like to test.

Definition: A/B Testing

-This test is sometimes known as split testing, is a designed experiment used to consider how a small difference in a digital marketing campaign might change customer or user behavior. It is a test used to compare two or more versions of something to figure out which one performs better. -The process of testing two web elements to find out which performs better.

If you wanted to focus on ad placement for your display ad, you could test for some of the following:

-Whether the ad performs better next to a particular type of content -What sites the ad does best on -What targeting settings might make the ad perform better

You might also want to test the creative aspects of a display ad. For example, you might change one of the following:

-the ad copy -the graphic or image -the size of the ad

What are the steps of the experimentation process?

1. Determine what needs to improve 2. Hypothesize change 3. Identify variables and create variations 4. run experiment 5. measure results 6. iterate

What are the main elements to test for in your ads?

1. Headline 2. Text in the body of the ad 3. Link to the landing page or the product page 4. Keywords the ad displays for

What elements can be A/B tested?

1. Logos 2. Email Subject Lines 3. Landings Pages 4. Call to action buttons 5. Any web communication element

Examples of dependent variables in an A/B test:

1. Number of return visitors 2. Number of booked flights 3. Conversion rate from a free to a premium product

What are some specific examples of of independent variables in an A/B test?

1. Two colors of a button (green/blue) or other element 2. Font size of the text in an ad 3. Keywords

A/B testing can help increase which three of the below options? 1. User engagement 2. Bounce rate 3. Conversion rate 4. Risk 5. Sales

A/B testing can improve user engagement, conversion rate, and sales. It can help reduce bounce rate and risk. -Note that bounce rate measures how long a user spends on the site before "bouncing" (leaving the page). You want a low bounce rate—in other words, you want users to stay on the page as long as possible before leaving. The longer a user stays on the page, the more chance you have of getting the user to take action.

What steps are involved in the scientific method?

Drag the steps into the correct boxes. 1. Make an observation and ask a question. 2. Form a hypothesis. 3. Make a prediction based on the hypothesis. 4. Test the prediction 5. Analyze results. 6. Use the results to make new hypotheses or predictions.

Definition: Variables

These are the actions and values that represent these causes and effects are called variables; they are the things that are changing in your experiment.

Why are experiments typically performed?

These are typically performed for cause-and-effect relationships.

What do cause-and-effect relationships typically explain?

They explain why things happen and allow you to reliably predict what will happen based on your actions.

Definition: independent variable(s)

This is the one or ones that are changed by the experimenter. You will manipulate one or more independent variables to determine how a change in the independent variable(s) affects the dependent variable. Usually this is a quality like colors, fonts, and design versions of various page elements. In other words, these are usually things that aren't measured with numerical values.

Definition: dependent variable(s)

This is what we are observing and measuring. It is called the dependent variable because we are trying to figure out whether its value depends on the value of the independent variable. Essentially we're trying to figure out what the cause-and-effect relationship is between this value and the independent variable, or if any relationship even exists. The dependent variable should be numeric data, like clicks, views, or another metric.

Definition: iterate

This means to continuously build and rebuild something until you get to a version that works best. Iteration is why there are 10 versions of the iPhone. Iteration is an important part of business and marketing because it means that a company is continuously improving its product or marketing efforts to better reach customers and meet their needs.