A/B Testing

Optimizely user roles

Administrator, Project Owner, Editor, Viewer

Types of goals

Click goal, page view goal, custom goal, revenue goal

Statistical significance

Statistical significance represents that likelihood that the difference in conversion rates between a given variation and the baseline is not due to chance.

Multi-page testing common uses

Testing different design directions against one another can easily be done using multi-page testing.

Multi-page limitations

Tests with too many variables take longer to run; it will also be more difficult to determine the impact of each individual change you make to each page. When setting up a multi-page test you must have the same number of variations for every page that is part of the experiment.

REST API

The Optimizely REST API lets you create and manage Optimizely projects and the experiments inside of them.

Multivariate test common uses

The most commonly cited example of multivariate testing is a page on which several elements are up for debate — for example, a page that includes a sign-up form, some kind of catchy header text, and a footer.

Multivariate test limitations

The single biggest limitation of multivariate testing is the amount of traffic needed to complete the test.

Disadvantage of using GTM to add Optimizely snippet

GTM loads asynchronously, causing variations to "flicker" before loading.

Conditions for Optimizely to work with GA

In order to integrate Optimizely with Universal Analytics, you must have an available Custom Dimension (note: this refers to a Custom Dimension in Google Analytics, not the Optimizely feature) to populate with Optimizely experiment data. Google Analytics integration will not function properly unless the Optimizely snippet is above the GA snippet.

Types of A/B testing

3 types: split test (normal A/B testing), multivariate test, and multi-page test

A/B Testing

A/B Testing is comparing two versions of something to see which one performs better.

A/B Test (Split test)

A/B Testing, also known as split testing, is a method of website optimization in which the conversion rates of two versions of a page — version A and version B — are compared to one another using live traffic.

Split test limitations

A/B testing will not reveal any information about interaction between variables on a single page.

Term - Thought - Something You've Learned

Definition - Answer - ETC

Multi-page funnel testing

Multi-page (also known as "funnel") testing is similar to A/B Testing except that rather than making variations to a single page, the changes you make are implemented consistently over several pages.

Multivariate testing

Multivariate testing uses the same core mechanism as A/B testing, but compares a higher number of variables, and reveals more information about how these variables interact with one another. Think of it as multiple A/B tests layered on top of each other. The purpose of a multivariate test, then, is to measure the effectiveness each design combination has on the ultimate goal.

Split test common uses

One of the most common ways A/B testing is utilized is to test two very different design directions against one another. A/B testing is also useful as an optimization option for pages where only one element is up for debate.

URL targeting

Simple match, exact match, substring match, regex

Dimensions

Visitors to your website are not all the same. They differ by source, browser, and other important attributes. Optimizely stores data about every visitor to your page, like when they arrived, what device they use, and where they came from. Each type of information that Optimizely collects is called a dimension.

Why integrate Optimizely and Google Universal Analytics?

You'll have a direct view of how Optimizely experiments affect the metrics you track in Google Analytics. You'll be able to see Google Analytics data for each variation in your experiment. You'll be able to filter your Google Analytics reporting by visitors who were successfully included in an Optimizely experiment and exclude those who weren't.

When to use Manual Activation Mode

You're easily able to deploy the Optimizely API call code to your site and would prefer calling activate from within your native JavaScript for finer-tuned control Your site is set up such that all experiments need to be manually activated at some time after Optimizely loads A very specific action must be taken on the page to activate an experiment, and that action can only be captured from within your site's native code

Do you need Manual/conditional activation?

You're testing a multi-step form, which appears in a modal and uses a "Next" button but doesn't load a new page. You're testing a web application, which re-renders the page when visitors perform certain actions, but doesn't re-load the page. You're testing an e-commerce site and want to run an experiment when customers modify the product. You're running an experiment that changes an object that appears at a certain scroll depth. You need to run experiments on an AJAX site, where elements are dependent on ajaxComplete You're running an experiment on a site using Angular, Backbone, Ember, or Knockout.

When to use Conditional Activation Mode

Your site is built on a single page app framework (like Angular.js or Ember.js) and you want to activate an experiment based on an event fired by that framework. A visitor must take certain actions on a page before being bucketed into an experiment, such as triggering a modal, scrolling to a certain point, or activating a widget or other dynamic module There is an AJAX request that happens long after the page has finished loading, and that AJAX request returns new content that needs to be modified