Business Intelligence MIS 5342 Baylor

Define business analytics

The extensive use of data, statistical and quantitative analysis, explanatory and predictive models, and fact-based management to drive decisions and actions. BA enables BI.

what does rebinning do?

breaks down a bin into a more manageable view- shows a more normal distribution- can help see anomolies

What is the business analytics challenge?

Getting anything useful out of tons of data

Select the Measurement Level

qualitative- quantitative- ordinal- binary

What is a chain count in sequence analysis

the number of items in a sequence- default is 3- max is 10

most likely association

they largest transaction count

Confidence

A probility based on a condition. Given A- what is the likelihood of B.

Association Analysis Support:

Frequency of the item set- Ex. These items occur together in 'basket' 54% of time

Supervised classification

The model predicts a target value- you have some idea what you are looking for

What is dynamic ticket pricing and how has it benefited the Giants?

The price of a ticket goes up or down depending on demand. Giants sold out 230 games in a row

What is the financial impact of data quality problems?

" Inability to: track customers ID valuable customers ID selling opportunities Market effectively Track revenue due to inaccurate invoices Build strong customer relationships"

Pattern Discovery Caution

" Poor data quality -missing- incomplete- outdated data- erroneous Opportunity (law of large numbers) Intervention (observation can change behavior) Separability Obviousness (pregnancy occurs in females) Nonstationarity (distribution changes over time)"

Sequence Analysis

" What items go together over time If a customer has X- what product is he most likely to get next? Next-best-offer campaigns"

What is data quality? Traits?

"""they are fit for their intended uses in operations- decision making and planning- or correctly represents the real world it is portraying"" Accuracy ◦ Completeness ◦ Consistency ◦ Uniqueness ◦ Timeliness"

What are traits of IVs used in clustering?

"""• Meaningful to the analysis objective • Relatively independent (not highly correlated) • Limited in number • Interval level (quantitative- continuous measure) • Number that is meaningful when math is applied. • Variables have to be standardized. • Low skew and kurtosis (symmetrical/bell-shaped distribution)"""

SAS Category

"A data item whose distinct values are used to group and aggregated measures. There are five types of categories: alphanumeric- date- datetime- time- and numeric. Alphanumeric categories can be made up of all letters- all digits- or a combination of the two. Categories that have values that are all digits might be physically stored as character or numeric data. The data type affects how values are handled in relation to some functionality- such as filtering- sorting- and formatting. Examples of alphanumeric categories include data items such as Product ID- Country- Employee Number- and Employee Name. Alphanumeric categories sort lexically. Date- datetime- time- and numeric categories are sorted by their underlying numeric values. Category data items can also be numeric. A category data item sorts differently than an alphanumeric data item. Numeric category data items sort by number."

SAS Measure

"A data item whose values can be used in computations. These values are numeric. Examples of measures include Sales Revenue- Units Sold- and Salary. The designer assigns a default aggregation method to every measure. Almost all measures are assigned sum. You can change the aggregation method. The process of assigning numbers to things such that the properties of the numbers reflect some attribute of the things. "

What is a solution for skew?

"Apply a math rule to all the data points to transform the distribution so that it is approximately normal (bell-shaped). Square each data value Take log of each value "

Categorical- ordinal- interval- binary

"Categorical/Nominal: Gender or Hair Color Ordinal: Similar to nominal but you can order it (low- medium- high) Interval: numeric (dollars) Binary: 0/1 such as Y/N"

How to select the number of clusters?

"Choose # clusters consistent with your objectives. Classify potential owners and non-owners of riding lawn mowers- then at least k = 2 clusters. • Promoting 3 types of discount offerings- then at least k = 3 clusters. 1. Convenience - it is convenient to market to just 2 or 3 groups. 2. Constraints - you have 6 products and you want 6 groups"

K means clustering

"Cluster Node IVs are standardized (by default) Select # of clusters (User Specify in properties panel) Default is 50 clusters and reduce using Cubic Clustering Criterion (CCC)"

Cluster analysis

"Cluster analysis is an exploratory tool. Useful only when it produces meaningful clusters. Be wary of chance results; data may not have definitive "real" clusters"

Correlations - small- large- pos- neg

"Correlation: a mutual relationship or connection between two or more things. Small: when A occurs- B will happen some of the time <0.5 Large: when A occurs- B will happen most of the time- >=0.5 Positive: When A goes up- B goes up Negative: When A goes up- B goes down"

What are some sources of poor data quality?

"Data entry by employees Data entry by customers Changes to your source system Data migration or conversion projects Mixed expectation by users External data Systems errors"

Ward's Method

An algorithm for hierarchical cluster analysis. In this method each observation is considered a cluster, & the clusters are hierarchically joined, based on minimizing the ratio of the varation between clusters to the variation witin clusters. Based on a statistcal analysis the # of clusters is selected. This number of cluserts is used for k-means cluster analysis.

Why are data sometimes transformed?

Conversions or integrations between systems- standardization

Correlation vs causation

Correlation does not imply causation means that a correlation between two variables does not necessarily imply that one causes the other.

Who were Harrah's 'most valuable customers'

" 26% of players brought in 82% of revenues (80/20 rule). These customers were not VIP's- which was the expected outcome. Instead they were middle aged and senior people from all walks of life- who had some discretionary income and extra time they could spend playing. The other key was that these customers generally played slots- not table games. "

IHOP downstream implications of bad data?

"a. Costs are incorrect for the product b. Sales are not apples to apples since the product is different c. Customer confusion between regions "

What are the ranges for the correlation coefficient?

+1 to -1

BA vs BI

1. Business analytics is an integral part of BI. "I think of analytics as a subset of BI based on statistics, prediction and optimization. The great bulk of BI is much more focused on reporting capabilities. BA is slowly replacing BI in many instances." 2. BI and BA are synonymous. "The term business intelligence is used by the information technology community, whereas business analytics is preferred by the business community. The two terms are synonymous and will henceforth be referred to as BI/BA." 3. BI and BA have key differences. Business intelligence describes: "What happened?" Business analytics describes: "Why did it happen?" (Statistics - explanation) "What will happen?" (Prediction) "What is the best that can happen?" (Optimization)

What is the difference between association and sequence analysis?

Association analysis looks at which items are likely to be purchased together- such as a market basket. Sequence analysis looks at the order the items are purchased to determine what a customer is likely to purchase next.

What is cross-selling?

Attaching an additional product to an existing sale.

Optimum number of variables in clustering

Clustering does best with fewer variables, not hundreds.

Example of a cross-sell

If product A and B are often purchased together- putting them on sale at different times can drive purchases continually.

Example of an up-sell

If product A and product B often go together- then placing a more expensive B alternative near the display for A can create an up-sell opportunity

What was the general strategy (prior to 1990s) for generating revenue in the casino industry?

Incentives like free rooms and meals- flashy premises (volcanos- ship battles- fountains)- and amenities (shows- spas- buffets).

• What measure assigns records to clusters?

K-cluster

Perfect correlation

Perfect positive correlation (a correlation co-efficient of +1) implies that as one moves- either up or down- the other will move in lockstep-in the same direction.

Predictive modeling

Supervised: linear regression- logistic regression- decision trees

Excellent visualizations

Tell a story- know your audience- present data objectively- present facts straightforward manner- don't censor negative data or data that does not fit- explain how to read it- use right type of visual- pleasing colors and good mindset- avoid lie factor- are you presenting or circulating? presenting is clear/concise and easily seen at a distance- circulation is more detailed/more notes to explain

What do Lift values less than 1 indicate?

That the association is less likely than the average.

What is 100% confidence?

The rule always occurs

Why did the Insurance Company (IFA) want loyalty card information from ShopSense?

To find out more about their customers by drawing correlations from their grocery habits to help forecast future events that would impact insurance payouts- both positively and negatively.

Discovery

Unsupervised: Cluster analysis- association rules

Census Data: Variable Roles

Use Input role for variables of interest

Sequence Analysis

What item follows after another item is bought. Across time

What are some methods used for undirected data mining?

clustering- sequence analysis- market basket- profiling

What is a data broker

companies that collect consumers' personal information and resell or share that information with others

Correlation matrix

darker colors show stronger correlation

Successful application of pattern discovery

data reduction- novelty detection- profiling- market basket- sequence analysis

How is Sam's Club implementing BI?

eValues- targeted at their Plus customers- customized coupons- based on their buying history- and by doing so- increases both loyalty and purchases made by these customers. The program focuses not only on encouraging repeat sales- but also uses a predictive element to encourage sales of items that Sam's Club thinks they might purchase- given sufficient incentive.

Ways pattern discovery can fail

poor data quality- opportunity (law of truly large numbers)- intervention (fraud detection)- separability of the interesting from the mundane- obviousness (equal number of married men and married women)- nonstationarity (things change and historic data is now irrelevant)

eValue success indicators

response rate for the ecoupons. Normally the response rate is 1% to 2%- but the targeted ecoupons saw usage rates from 20% to 30%.

Association analysis

what items are purchased at the same time?

types of data

Qualitative and quantitative

What are data quality problems?

Missing data- duplicates- incomplete- inaccurate

How is data transformed?

"ETL -Extraction- transformation- and loading ◦ a process that extracts data from internal and external databases- transforms the data using a common set of enterprise definitions- and loads the data into a data warehouse."

benefits to ShopSense in selling its data to IFA.

"Financial Access to IFA's analysts If ShopSense was open with customers about what it did with their data- it might build trust If ShopSense retained control of the data and any programs were jointly marketed with IFA- it could provide ShopSense customers unique offerings and enhance loyalty- thus driving sales Group insurance plan costs might be lowered. "

risks to IFA in using ShopSense data?

"If customers found out they might go to a competitor False correlations (people buying for others) It would cause issues if the company wanted to operate in other countries with stricter privacy laws "

ETL

"In a data warehouse - ETL -Extraction- transformation- and loading ◦ a process that extracts data from internal and external databases- transforms the data using a common set of enterprise definitions- and loads the data into a data warehouse."

reasons about why mining patient's data is a good idea and 2 reasons why this might not be a good idea.

"It can be helpful for the patient's health to have indicators in place to proactively move them towards healthier behavior. In some cases- it may save a life.Specific things such as pharmacy reminders are useful- less invasive tools that will help patients without causing privacy concerns. Looking at patients' personal habits to predict health is a very slippery slope. The data does not know who the purchases are for (the patient or someone else) and inaccurate data may become attached to the patient's record. The privacy concerns here are very troubling. Many people do not realize their purchases are tracked by data brokers and would not want to share their information if they did- particularly with a healthcare agency. "

What algorithm does SAS use to cluster?

"K cluster - Euclidean distance from each case in the data to each cluster center is calculated. Each case is assigned to closest cluster center. 1. Select variable inputs. 2. Select k cluster centers. 3. Assign cases to closest center. 4. Update cluster centers. 5. Re-assign cases. 6. Repeat steps 4 and 5 until convergence."

Lift

"Lift is a general measure of association between the A & B. A Lift of 2 means a customer is twice as likely to purchase B as the average. Lift indicates how efficient the rule is in finding a real association compared to random selection. Lift values > 1 indicate positive correlation between A & B. Lift values < 1 indicate negative correlation between A & B. Lift values = indicate no correlation between A & B. A lift ratio greater than 1 suggests some usefulness to the rule. The larger the lift ratio the greater the strength of the association. "

Why do we care about shape of distribution?

"Many statistics inferences require that a distribution be normal or nearly normal Normal = bell shape"

Measurement Level

"Nominal = qualitative- catagorical Interval = quantitative- continuous Binary = 0- 1"

What 2-3 factors have contributed to the data deluge?

"Prevalence of automatic data collection Electronic instrumention OLTP Online transaction processing"

sequence analysis confidence

"Probability of getting 'B' next- after getting 'A'. If customer has 'A'- prob of getting 'B' next. "

Why is standardization important in cluster analysis?

"Problem: Raw distance measures are highly influenced by scale of measurements ($- %- psi- sq feet etc.) Solution: normalize (standardize) the data first Subtract mean- divide by std. deviation to get rid of units ($- % etc) Also called z-scores "

best ways to meet analytical challenges in an org

"Recruiting internal talent Hiring consultants Training "

What is purpose of the filter node?

"Remove problematic cases- missing records- errors Remove rare cases Outliers Default is to remove any cases with inputs > 3 Std deviations from the mean."

What is Ward's Method in SAS?

"SAS default (Ward's Method) - Begins with 50 cluster centers - Hierarchical clustering is performed and the best number of clusters is based on the CCC - cubic clustering criterion. - # of clusters corresponds to the peak in the CCC plot"

What is clustering?

"Segmenting the data. Form groups (clusters) of similar records (data records represent customers) 1. CLUSTER: Group customers into distinct segments based on demographic variables and/or behaviors 2. PROFILE: Create customer profiles or descriptive tags • Use for future marketing strategies. Use for segmenting markets"

Variables in clustering

"Select variables that are meaningful to the clustering project. - Give them the Role of Input"

Skew

"Skew: a measure of the asymmetry of the probability distribution of a variable about its mean- when data points cluster more towards one side or the other on the scale left: negative skew- left tail is longer & moves towards lower values right: positive skew- right tail is longer & moves towards higher values positive: scores fall towards the lower side with few higher scores- the mean is usually greater than the median- greater than the mode- long tail to the right negative: scores fall towards the higher side with few low scores- the mean is usually less than the median- less than the mode"

Standardization

"Standardize the variables to eliminate the units of measure since k-means uses Euclidean distance to cluster records Standardization of the IVs is the default in SAS"

market basket

"Strength of the association between two or more products: Support - frequency of the item set Confidence - if 'basket' contains A- what is the probability it also contains B. Lift Ratio - general measure of association"

Lift- Support- & Confidence

"Support: probability that a transaction will include A&B. Support is the frequency of the item sets Confidence: conditional probability - it the basket has A- what is the probability it will also have B Lift: the confidence divided by the expected confidence. expected confidence is the probability of B"

How to measure the strength of an association?

"The Strength of the association is measured by the support and confidence of the rule. In the Association Report- they are ordered by strength. The strongest rule will be the first. The weakest rule will be last."

Why was it a good idea to financially motivate Harrah's employees to provide good service?

"This offered an incentive for staff to improve not only their own service performance- but to become a team and insure that others were also providing friendly and speedy service. The company paid out customer service bonuses based on satisfaction scores- not on profitability. "

Profile Node

"To describe the characteristics of the records comprising each clusters. HH size HH income Reg Density"

correlations in the ShopSense data

"Trans fats and heart disease Quarterly (or more) purchases of pistachios and bananas showed minimal cases of MS or Parkinson's Sales of prophylactics and HIV claims "

top 3 analytical challenges

"Uncertain ROI Lack of analytical talent Business-IT Gap "

Cluster analysis

"Used for pattern discovery. Purpose is to group data records based on similarities of the input variables (IVs). "

What is 'proactive care management' and what does it have to do with patient data?

"Using models (based on the consumer data)- the organization can predict who may be at risk for various issues from diabetes to depression to asthma to being late to office appointments. By identifying these patients- the organization can take steps to mitigate the problem- such as with a nurse's call or pharmacy reminder to refill a prescription. "

Four Visualization Concepts

"We don't notice everything we see- but we can easily identify patterns we know to look for. We don't pay attention to everything we see- but our eyes are drawn to contrasts. We don't remember everything we see- only that to which we attend. Our perceptual abilities are extraordinary- but limited by working memory. - Encode info into chunks of memory. - Side-by-side data comparison"

Data warehouse

"a logical collection of information. ◦ gathered from many different operational databases ◦ supports business analysis activities and decision-making tasks The primary purpose of a data warehouse is to aggregate information throughout an organization into a single repository."

data steward

"a. Enforced data consistency b. Enforced participation c. Decided where data lived d. Maintained accurate reporting methods "

IHOPs' solutions

"a. FRED Franchise and Restaurant Enterprise Directory b. Data steward c. Executive oversight d. Holding people accountable e. Standard POS f. Training g. Data sharing with franchisees "

Why are data consistency and quality important?

"a. Garbage in garbage out b. The results are not useful because they are not a true indication c. Bad data can skew results and encourage managers to make bad business decisions "

IHOP data problems

"a. Missing data b. Incorrect data c. Duplicate data d. Inconsistent data "

Unsupervised classification

"cluster and profiling- and market basket and sequence analysis Unsupervised: No target variable (DV) with which to associate other variables. Not a predictive method. - no DVs- no predictions - Purpose is to group data records based on similarities of the input variables (IVs)"

traits of a meaning maker

"integrate analytics into regular work processes more effective at using analytics tools create special teams focused on analytics receive analytics benefits in a minimum of 5 out of 10 key areas have the right systems/infrastructure and tech experts to make it happen hire software developers to design their analytics software expect significant revenue gains from analytics make significantly higher profits "

What does skew do to the mean

"positive: - the mean is usually greater than the median- greater than the mode- long tail to the right negative: the mean is usually less than the median- less than the mode- tail to the left"

What are the characteristics of Big Data?

"volume- velocity- variety- variability- complexity So large it cannot be processed using conventional methods Structured (bank transaction) and unstructured (tweet) "

Select Input Variables

"• Meaningful to the analysis objective • Relatively independent (not highly correlated) • Limited in number • Interval level (quantitative- continuous measure) • Number that is meaningful when math is applied. • Variables have to be standardized. • Low skew and kurtosis (symmetrical/bell-shaped distribution)"

Characteristics of the data warehouse:

"◦ Organization. Data are organized by subject ◦ Consistency. Data are coded in a consistent manner ◦ Time variant. The data are kept for many years so they can be used for trends- forecasting- and comparisons over time. ◦ Nonvolatile. Once entered into the warehouse- data are not updated. ◦ Relational. Typically the data warehouse uses a relational structure. ◦ Client/server. The data warehouse uses the client/server architecture mainly to provide the end user an easy access to its data. ◦ Web-based. Data warehouses are designed to provide an efficient computing environment for Web-based applications"

Measurement Scale

$- %- count

How to calculate Lift

(Confidence [A=>B]) / (Expected Confidence [A=>B]) Expected confidence = (all transactions containing B/all transactions)

How to calculate Confidence

(transactions that contain every item in A & B) / (transactions that contain the items in A). Conditional probability of B given A.

How to calculate Support

(transactions with A and B) / (all transactions) it is a percentage and is symmetric because A=>B is the same Support as B=>A.

What is the average revenue gain and improvement in cost reduction when a corporation uses analytics for decision making?

8.4% increase in revenues and 8.1% improvement in cost reductions

What is the difference between a data set row and column?

A row is a record or observation. A column is a specific variable for that record

k-clustering

Algorithm for clustering- k specifies the number of clusters. k-means is one of the simplest unsupervised learning algorithms that solve the well known clustering problem. The procedure follows a simple and easy way to classify a given data set through a certain number of clusters (assume k clusters) fixed apriori. The main idea is to define k centers- one for each cluster. These centers should be placed in a cunning way because of different location causes different result.

Banking Services Case Study plan:

Association analysis (market basket)

Why is data quality important?

Business decisions are only as good as the quality of the data used to help make the decisions

On the Rules Table, How are the rules ordered?

By Lift

Harrah's strategy

Database marketing and decision science based analytical tools made them more effective than competitors who used intuition. They also focused on service instead of flashy incentives. They used the data- ran marketing experiments- and honed their programs to combine incentives and service in a unique way. They used the data itself to suggest the marketing strategies- rather than vice versa. They also focused on the "lifetime" value of the customer. Another significant change was the use of tiered loyalty club membership- where different levels of customer spending designated highly visible rewards like avoiding the long line at the restaurant.

Types of Data Mining Analysis

Discovery and Predictive Modeling

What is the range?

Distance between the lowest and highest values

What is the purpose of the filtering node?

ELIMINATE THE DATA THAT IS THROWING OFF THE DATA

3 consequences of Big Data?

Every activity will eventually generate data. Every company will eventually need analytics. Every person will need analytics eventually?

Banking Services Case Study goal:

Explore associations between retail banking services/products used by customers.

Sequence Analysis Support

Frequency of the item sequence. Ex. 54% of time- customers get cking first and svg next- in that order.

Support: Symmetric - CKG->SVG and SVG->CKG

Frequency of the item set

What is the implication knowing what your customers did in the past?

If you know what customers did in the past- and can figure out why they did it- you can influence future choices

Purpose of sequence analysis?

It is useful for discovering what items go together in a specific order- such as purchasing a barbecue grill in June and steaks in July. This creates opportunities for targeting additional sales based on the prior item- or "best next offer" campaigns.

CCC Plot

Negative CCC is not useful- the plot can help guide you towards selecting the optimal number of clusters

"How is the 'couponing model' changing? "

Personalization is taking coupons to a new level of driving purchase behavior. Sam's Club's eValues- with their custom printed coupons highlighting past purchases and likely future purchases- has been highly successful and will inspire competitors to do likewise.

Association Analysis Confidence

Prob a customer has 'B'- given he has 'A'. If customer has 'A'- prob of having 'B'.

Purpose of profiling

Profiling allows us to understand the traits of the records in each cluster

Qualitative vs quantitative

Qualitative - words- vs. quantitative - numbers

Difference between reporting and analysis?

Reporting: Is Provided Leaves you with questions rather than answers Is a Statement rather than a conversation Analytics: Is Presented Makes recommendations Is controversial, conversational Forces the Org to make a decision

the business analytics that drives eValue

Sam's Club analyzes customer purchase history in great detail at the micro level. The company found over a trillion permutations with Plus customers. particular keywords- such as "organic" or "environmentally friendly-" the timeframe in which purchases are made- the location- and which discounts- look at when they purchase and time coupons

What is SEMMA?

Sample- Explore- Modify- Model and Assess

What is up-selling?

Selling a new product to an existing customer

What bank products go together?

Sequence analysis

RegDens of 100

THE HIGHEST POSSIBLE DENSITY A REGION CAN BE (SCALE IS 1-100)

Supervised vs Unsupervised data mining

The difference is that in supervised learning the 'categories' are known. In unsupervised learning- they are not- and the learning process attempts to find appropriate 'categories'.

Carolinas HealthCare using Big Data and Predictive Analytics?

They are using consumer data about their patients to predict healthcare issues based on purchasing habits.

Association rule discovery

This is market basket analysis or affinity analysis. Consists of 2 variables: a transaction and an item. For each transaction there is a list of items.

Define Business Intelligence

Umbrella term, Inter-relationships among facts (variables), fact-based decision making, historical, reporting, used for business analytics. BA enables BI.

Association Rule

Used in market basket, it is a statement of the form (A=>B). The aim is to determine the strength of the association rules among a set of items.

Segment Profile Node

Variables in each cluster arranged by importance. 1st is most important to the cluster.

Why does a lift value of less than 1 have no value for market basket analysis?

We only care about groupings that go together (significant or interesting)

market basket

What items go together at a rate greater than a random association. One point in time

sequence analysis

What items go together from one time to another (not in same basket). This can create opportunity for best-next-offer campaigns.

Confidence

Will Not be the same for symmetric rules

Scatterplots

a graph in which the values of two variables are plotted along two axes- the pattern of the resulting points revealing any correlation present.

What is Market Basket analysis?

a selection of items purchased together

Data mart

contains a subset of data warehouse information ◦ Generally for a specific business unit or function

Correlation

mathmatical relationship between factors

What is purpose of profiling clusters?

descriptive tags for future marketing- use for segmenting markets

Business Intelligence - past- what happened

focuses on using a consistent set of metrics to measure past performance and guide business planning. Business Intelligence consists of querying- reporting- OLAP (online analytical processing)- and can answer questions including "what happened-" "how many-" and "how often."

What is data standardization?

insuring all terminology is the same- formats are the same- abbreviations are the same- etc. Data must be put in a standard format.

The objective of clustering

is pattern discovery

Purpose of a histogram

look at distribution

purpose of a histogram of a variable?

look at the distribution- PICK OUT ANOMOLIES AND TRENDS-

Interpret sequence rules

look at the table

Interpret the association rules

look at the table

What is Advanced Business Analytics?

the current state-of-the-art in the field. BI has a more query or reporting flavor. What happened yesterday? Advanced business analytics is forward looking. Why did it happen? What will happen next? What is the best decision?

Least likely association

the smallest transaction count

Business Analytics - future- what will happen

the study of data through statistical and operations analysis- the formation of predictive models- application of optimization techniques and the communication of these results to customers- business partners and colleague executives. Business analytics (BA) is a term that refers to the tools and techniques used to systematically examine any aspect of a business' performance.

Data Exploration: Why re-bin a histogram?

to break it down into more meaningful groups- identify anomolies- try to get more normal distribution

Why do Market Basket analysis?

to predict the likelihood of purchasing B if you buy A

What is undirected (unsupervised) data mining?

when you don't have a specific target in mind