Marketing Analytics Exam 2

which is true a. an experiment has 1 treatment only b. an experiment has 1 outcome only c. an experiment has 1 control variable only d. an experiment has 1 type of unit to which its treatment is assigned only

d. an experiment has 1 type of unit to which its treatment is assigned only

consider the following scenario. All chain restaurants had to put calories on their menus in the US by December 2016. Researchers measured their receipts the month before and the month after this happened to test whether their calories or menus effectively reduced overall calories consumption and/or reduced restaurant sales. This was an example of what kind of study a. randomized lab experiment b. natural lab experiment c. randomized field experiment d. natural field experiment

d. natural field experiment

which is considered the gold standard for determining causal inference a. interactive designs b. field experiments c. natural experiments d. randomized experiments

d. randomized experiments

which is false about the causal inference of x on y a. x must occur before y b. there must be evidence of an association between x and y c. control of other causal factors d. x must be manipulated

d. x must be manipulated

history effect

events external to the experiment that affect the responses of the people involved in the experiment

experiments-procedure

- decide on your treatmnets what are you trying to test? how many factors? what will you do if you have the information -design the experiment design -recruit participants and run the experiment -analyze the experiment data

the 41 shades of blue test at google

-a team at google couldnt decide between two blue -they test 41 shades between each blue -showing each one to 1% of their visitors to see which ones perform better

machine learning

-an application of artificial intelligence that allows systems to automatically learn and improve without actually being programmed to do so -machine learning focuses on creating systems that can access data and automatically apply themselves -as the data system gets more and more experience, the better it will become at analyzing data

AB tests

-are field experiments in the digital marketing context making use of big data -AB tests are often run as part of a live, real marketing effort -AB are often automated to be adaptive -are rooted in experimental design -however, often: AB test oftne test two or more competing messages without necessarily seeking to understand why, experiments test hypothesis-> seek to understand outcomes and reasons

before and after

-as the name suggests, with after only experimental designs measures the independent variable are only taken after the experimental subjects have been exposed to the independent variable - the before and after design is based upon the after only design, in that the effect of the independent variable, if any, is established by observing differences between the value of the dependent variables before and after the experiment

Lets pretend we work at facebook

-ask question: how can be stop more people from deactivating their facebook -do research: the reason why people use facebook is that their friends are on facebook -hypothesis: if you remind people about their friends who are being left behind, we will lover the deactivation rate

The Scientific Method

-ask questions -hypothesis -research -analyze data -conclusion

causal inference

-causal inference is the formal name for cause and effect interpretations of research methods like experiments -experiments yield higher causal inference with other analyses such as regression yield lower causal inference (ie they are more correlational)

comparison visualizations

-comparison charts are used to compare the magnitude of values to each other and can be used to easily find the lowest and highest values in the data -it can also be used to compare current values vs old to see if the values are increasing or decreasing - common questions are "what products sell best" and "how are our sales compared to last year"

composition visualizations

-composition charts are used to see how a part compares to the whole and how a total value can be divided into shares -a composition charts shows the relative value, but some charts can also be used to show the absolute difference -the difference is between looking at the percentage of total and value of total -common questions are "how big part of the market do we have in a region" or "what areas is our budget divided into"

examples of machine learning

-computational finance, for credit scoring and algorithmic trading -image processing and computer vision, for face recognition, motion detection and object detection -production, for price forecasting -automative, aerospace, and manufacturing, for predictive maintenance -natural language processing for voice interaction

distribution visualization

-distribution charts are used to see how quantitative values are distributed along an axis from lowest to highest -looking at the shape of the data a user can identify characteristics such as the range of values, central tendency, shape and outliers -it can be used to answer questions such as "number of customers per age group" or "how many days late are our payments"

4 cornerstones of a data visualization

-distribution of a single variable -relationship between two variables -composition of a single or multiple variables -comparison between different categories/individuals

data visualization

-each time you map data to visualization, the key thing you need to consider is: what are you trying to show the user? it isnt a question of what looks pretty -learn about the right type of chart: you want a chart that demonstrates value and achieves its purpose in an easily recognizable way

interactive design

-introducing more than one factor typically yields rich insights -moderator variable: changes the way the independent variable affects the dependent variable -interaction: test of one factor across all levels of another factor

segmentation and target with AB testing

-many software systems allow marketers to analyze the success of options during the exploration stage of AB testing across different segments for applying the results intelligently across segments in the exploitation stage -this method could also be useful for determining which segments to target, especially if one option tested is more profitable than the other

after only

-measures of the independent variable are only taken after the experimental subjects have been exposed to the independent variable -this is a common approach in advertising research where a sample of target customers are interviewed following exposure to an advertisement and their recall of the product, brand or sales features is measured

lab experiment cons

-more artificial -physical environment -small data -more control over other influences

AB testing pros

-more realistic -digital environment -big data -less control over other influences

AI marketing: content personalization

-organisms amplifying group intelligence by forming flocks, schools, shoals, colonies and swarms (eg, bees, birds, fish, ants) -research shows: groups outperform most individuals in decision making and prediction regardless of expertise -most surveys, focus groups, and polls fail to recognize this social aspect of decision making

Applied (most common)

-perform single task extremely well -run automated and repetitive tasks -does not involve decision making -ex: netflix, spotify, self driving cars

what are the tools to deliver those three marketing principles

-product -price -place -promotion

artificial applied intellegence

-refers to systems that are designed to work on specific tasks, such as trading stocks or controlling an autonomous vehicle -is most common

artificial general intelligence

-refers to systems that can handle any task -is more complicated since the systems are dealing with an infinite number of tasks rather than focusing on just one -is less common

relationship visualization

-relationship charts are used to see the relationship between the data and can be used to find correlations, outliers, and clusters of data -common quesions are "is there a correlation between advertising spend and sales for our products" or "how does expenses and income vary per region and whats the deviation"

causal inference in simple terms

-requirements for causal statement x->y 1. x must occur before y 2. there must be evidence of association between x and y 3. control of other causal factors -(1) is typically manipulated (2) can be assessed by the data (3) is typically done by randomizing , matching or blocking -product advertisement on tv-> choice of product purchased

2 categories of machine learning

-supervised -unsupervised

simple design

-the most basic experimental design has a single factor -a factor is an independent variable (referred to as x) that may affect the dependent variable (often referred to as y) -IV means the marketing input we are interested in changing -dv is outcome variable

Labratory vs field experiment (AB test)

-validity: a field experiment tends to have higher external validity but a smaller internal validity -exposure: by doing a field study, you may provide information to competitors or adversely affect the marketplace

marketing channels for AB testing

-website promotion (copy, images, video) -email campaign (messaging, subject line) -social media ads (messaging, images, video) -digital retailing ( messaging, pricing) -text coupons (messaging, price promotion) -mobile ads (messaging, iamges)

supervised machine learning

-where you ahve input variables (x) and output variable (y) and you use an algorithm to learn the mapping function from the input to the output (y) for

unsupervised machine learning

-where you only have input data (x) and no corresponding output variables -the goal for unsupervised learning is to model the underlying structure or distribution in the data in order to learn more about the data -algorithms are left to their own devices to discover and present the interesting structure in the data

AI marketing: tool platforms

1. albert -used for autonomous media buying -buying digital media on behalf of clients -analyzes, manages, and optimizes paid advertising campaigns 2. frank -pay per click -uses machine learning to find best paid advertising channels depending on specific audience

three things needed for causal inference

1. x must occur before y 2. there must be evidence that x is associated with y 3. other causal factors are ruled out

which is true about designing AB tests to extract maximum meaning a. A and B should have identical content except hypothesized elements b. A should always be in control c. A and B should have different content except for the hypothesized elements d. A and B should have two levels each

A. a and b should have identical content except for the hypothesized elements

which is not true about a natural experiment a. synonymous with a "lab experiment" b. observation of naturally occurring incidents c. sometimes the only option because conditions cannot be randomized d. not a true randomized experiment

a. synonymous with "lab experiment"

Exploitation stage

a. then the software determines which version was more successful b. you then tell the software algorithm what marketing outcome is important in order to test whether version A or B is more successful (click through rate, purchases) c. the winning version is exposed to the large remainder of the potential audience that was not part of the exploration stage

consider that a company wants to run an experiment to see which of the four advertisements is most effective for a new product presented with five distinct price point. People are randomly assigned to see one of the four advertisements and one of the five price points. How many factors are part of this design a. 2 b. 9 c. 1 d. 20

a. 2

which is true a. AB testing is only done in a digital environment b. AB testing is only done for websites and email c. AB testing is only done for direct mail d. AB testing is often done in brick and mortar stores

a. AB testing is only done in a digital environment

which is true a. AB testing is only done in digital environment b. AB testing is only done for websites and email c. AB testing is only done for direct mail d. AB testing is often done in brick and mortar stores

a. AB testing is only done in a digital environment

consider the after only experimental design. which of the following is TRUE a. an advantage of this design is that there is no pre-test bias b. a disadvantage of this design is that there is no pre test bias c. an advantage of this design is that it is possible to reduce post test bias d. an advantage of this design is that it is possible to measure pre test bias

a. an advantage of this design is that there is no pre test bias

exploration stage of AB testing

a. choose what portion of the potential audience to explore b. AB software randomly selects people to assign to the exploration stage

field experiments a. conducted in a natural setting b. conducted by practitioners in the field of concern c. conducted only in the digital world d. conducted only in an outdoor setting

a. conducted in a natural setting

AB tests are a. controlled experiments b. more similar to lab experiments than field experiments c. never field experiments d. not truly experiments

a. controlled experiments

which of the following it true about after only experimental design a. it satisfied come conditions for causality b. it satisfies all conditions for causality c. it satisfies no conditions for causality d. it satisfies only one condition for causality

a. it satisfies some conditions for causality

which of the following is not necessary to determine that a marketing campaign causes an increase in sales a. we find that there are other external factors that affect sales but not the marketing campaign b. we find that when we launch a marketing campaign, there is an increase in sales c. we find that when we dont launch a marketing campaign, there is no change in sales d. we find that when we launch a marketing campaign, it increases sales later

a. we find that there are other external factors that affect sales but not the marketing campaign

AB testing

also called ABN testing or split testing

market

an exchange between two partners (frequently buyers and sellers but also apply to non-profit or non-monetary exchange)

experimentation

an experiment is a procedure in which one or more casual variables - for example, choice to buy the product or not-gathered while controlling the other variables that may influence the effect variable (Christmas season, weather, etc)

artificial intellegence

area of computer science in which machines have human like capabilities

attribution

assigning credit to the events that correctly lead to marketing conversions

AB tests are a. AB tasty b. AB lab c. Qubit d. google experiments

b. AB lab

which of the following is NOT an AB testing tool a. AB tasy b. AB lab c. Qubit d. google experiments

b. AB lab

which statement is false a. AB tests are often run as part of a live marketing effort b. AB tests= real marketing firms brick and mortar store experiments c. AB tests=split tests d. AB tests are often automated

b. AB tests= real marketing firms brick and mortar store experiments

consider the before and after experimental design. Which of the following is true a. an advantage of this design is that it is possible to measure pre-test bias b. a disadvantage of this design is that there is pre test bias c. an advantage of this design is that there is no pre-test bias d. an advantage of this design is that it is possible to reduce post test bias

b. a disadvantage of this design is that there is pre test bias

field experiments are... a. conducted by practitioners in the field of concern b. conducted in a natural setting c. conducted with field mice d. conducted only using emails or catalogs

b. conducted in a natural setting

which is true of a bandit test a. it is an AB test but with an extra hostage stage b. it has an adaptive exploitation stage c. it is an AB test with a hidden exploration stage d. it removes people from the exploitation stage

b. it has an adaptive exploitation stage

which is true of a bandit test a. it is an AB test with an extra hostage stage b. it has an adaptive exploitation stage c. it is an AB test with a hidden exploration stage d. it removes people from the exploitation stage

b. it has an adaptive exploitation stage

a company called beyhive launched a promotional campaign and wants to know if it was effective. Which of the following pieces of evidence does NOT support a conclusion that the beyhive promotional campaign caused and increase in sales a. no marketing campaign is followed by increases in beyhive sales b. none of these options c. launching the marketing campaign is followed by increases in beyhive sales d. launching the beyhive proomotional campaign today corresponds to beyhive

b. none of these options

which is true a. protect your AB test from too much input from other departments in the company b. people often use AB tests to test competing marketing ideas in a department c. AB testing is not a time to try crazy ideas d. the value gained from AB testing usually outweighs potentially frustrating customers

b. people often use AB tests to test competing marketing ideas in a department

which of these is an example of applied AI a. R2D2 b. spotify c. pac-man d. hadoop

b. spotify

what is the problem of local maxima a. AB tests fail test conditions that will optimize the companies local concerns b. AB tests fail to maximize the exploitation stage c. AB tests fail to test condition that may have better outcomes d. AB tests fail to attract local participants

c. AB tests fail to test condition that may have better outcomes

what is the problem of local maxima a. AB tests fail to test conditions that will optimize the companies local concerns b. AB test fail to maximize the exploitation stage c. AB tests fail to test conditions that may have better outcomes d. AB tests fail to attract local participants

c. AB tests fail to test conditions that may have better outcomes

an example of a randomized, controlled experiment which is not AB test is a. a quasiexperiment in the digital world b. a field experiment in the digital world c. a lab experiment in the non-digital world d. a natural experiment in the non digital world

c. a lab experiment in the non digital world

which AB testing stage typically uses a larger portion of the potential audience a. bandit stage b. experimental stage c. exploitation stage d. exploration stage

c. exploitation stage

which AB testing stage typically uses a larger portion of the potential audience a. bandit stage b. experimental stage c. exploitation stage d. exploration stage

c. exploitation stage

which of the following is not an AB test a. split tests b. controlled experiment in the digital world c. optimize tests d. ABN tests

c. optimize test

which of the following is not an AB test a. split tests b. controlled experiment in the digital world c. optimize test d. ABN test

c. optimize test

maturation effect

changes in the respondents that are a consequence of time, such as aging, getting hungry, or getting tired

selection effect

if units self select themselves into the treatment and control groups then this is of serious concern if the election reason is related to the outcome of interest

moderator variable

it changes the way the independent variable affects the dependent variable

weakness of AB testing

it has less control over other influences on the results than lab experimentation

independent variable

manipulated variable for which marketing analyst is interested in seeing an effect on dependent (outcome) variable

causation

means that one of these variables gives rise to the other behavior

place

refers to either the physical location where a business carries out business or the disutribution channels used to reach markets

promotion

refers to the marketing communication used to make the offer known to potential customers, such as advertising, public relations, direct selling, and sales promotion

price

refers to the total cost to customers to acquire the product (comes first)

product

refers to what business offers for sale and may include products or services

randomization

research participants are assigned to see different versions of x by chance, rather than by participant choice or researcher bias -randomization ensures that other causal factors are ruled out

pre-test effect

the fact that someone has been measured previously might affect their future behavior

instrument variation

the method used to collect data changes within the experiment (questionnaire, interviewer, etc)

dependent variable

the outcome variable

morality (drop out)

the sample becomes unrepresenative

correlation

two variables behave in some way similar to one another