ISM 4545 Exam 1

In information theory, _______ is a measure of uncertainty associated with a random variable. Homogeneity Entropy Gini P-value

entropy

Confusion matrix can be fully constructed, given the accuracy value of a classifier. T/F

False

It is not possible to use binary split for continuous attributes, since there are at least 3 distinct values. T/F

False

Visualization: Side by Side Box Plot

Side by side boxplots are very effective in showing differences in a quantitative variable across different levels

Types of Classification Methods

Discriminative -Decision Tree -Support Vector Machine -Nearest Neighbor Regression based -Logistic Regression -Artificial Neural Network Probabilistic -Naïve Bayes

precision equation

TP / (TP + FP) a / a + c

Limitation of Accuracy Metric

Simplistic

Assume that E depicts an event and A depicts an observation. P(E|A) is:

The conditional probability of E, given A

NN: When to Stop Updating?

-When weights change very little from one iteration to the next -When the misclassification rate reaches a required threshold -When a limit on runs is reached

Generalization

-We would want models to apply not just to the exact training set, but also to the general population -Unbiased

Assuming that "I" represents the entropy level of a set, I(3,6) ≠ I(6,3)

FALSE

Naïve Bayes Approach•Two assumptions:

1.Attributes are independent 2.Attributes are equally important

Overfitting:

the tendency to tailor a model to the training data as much as possible -Memorization -Confusing the noise in the training data (e.g. occurrences due to chance) with the signal (e.g. patterns)

Which of the following methods provide insights about the patterns in the training data? Naïve Bayes Decision Tree Artificial Neural Network All of the above.

Artificial Neural Network

NN: How a network learns?

-Backpropagation: The best-known supervised learning algorithm in neural networks -Learning is done by comparing computed outputs to desired outputs of historical cases

CRISP Process

1. Business Understanding (objectives/requirements) 2. Data Understanding 3. Data Prep 4. Modeling 5. Evaluation 6. Deployment

Confusion Matrix Components

A = TP B = FN C = FP D = TN

Data Exploration

A preliminary exploration of the data to better understand its characteristics Key motivations of data exploration include -Helping to select the right tool for preprocessing or analysis -Making use of humans' abilities to recognize patterns Especially useful in early stages of data analysis -Detect outliers (e.g. assess data quality) -Test assumptions (e.g. normal distributions or skewed?) -Identify useful raw data & transforms (e.g. log(x))

Methods can be compared and evaluated according to different criteria:

Accuracy Speed •Time to construct the model •Time to use the model Robustness •Handling noise and missing values Scalability •Efficiency in large datasets Interpretability •Understanding the model

In an artificial neural network, each input node represents a(n) Attribute Class Cluster Weight

Attribute

Two of the assumptions of Naïve Bayes are: Classes are independent and equally important Attributes are mutually exclusive and exhaustive Classes are mutually exclusive and exhaustive Attributes are independent and equally important

Attributes are independent and equally important

Methods of Estimation: Cross-Validation

Avoids overlapping test sets -First step: data is split into k subsets of equal size -Second step: each subset in turn is used for testing and the remainder for training

What is a Neural Network?

Biologically inspired computer models that operate like a human brain -These networks can "learn" from the data and recognize patterns

ROC (receiver operating characteristic)

Characterizes the trade-off between positive hits and false alarms (true positive rate vs. false positive rate)

Data Preparation and Preprocessing

Check the quality of the data Clean the data (if necessary) Transform the data (if necessary) Sample the data (if necessary)

Which of the following is a better split to use in a decision tree? Class 1: 5 Class 2: 5 Class 1: 4 Class 2: 4 Class 1: 2 Class 2: 2 Class 1: 7 Class 2: 8

Class 1: 7 Class 2: 8

Classification vs. Prediction

Classification- predicts categorical class labels Prediction- Predicts continuous-valued functions

Dataset:

Collection of data objects including their attributes

Quality of Data: Outliers

Data objects with characteristics that are considerably different than most of the other data objects in the data set

Which of the following is an example of a classification problem? Grouping customers of a bank into 3 segments based on customer value. Finding correlations between items sold at a grocery store Forecasting the monthly sale amount for the next 3 months at a retailer. Deciding whether a tumor cell is benign or not.

Deciding whether a tumor cell is benign or not.

One of the advantages of decision trees over artificial neural networks is that: Decision trees have better accuracy Decision trees don't face the problem of overfitting Decision trees reveal the pattern used for making the classification All of the above

Decision trees reveal the pattern used for making the classification

NN: Learning algorithm

Designing the network topology Sample cases are shown to the network as input and the weights are adjusted to minimize the error in its outputs

Naïve Bayes Discussion

Easy to implement It works surprisingly well (even if assumptions are not realistic) -Though, if there exist dependencies among the attributes, these cannot be modeled by Naïve Bayes Classifier Adding too many redundant attributes will cause problems

Quality of Data: Missing Values

Eliminate Data Objects -Estimate Missing Values -Ignore the missing value during analysis•

NN Advantages

Good predictive ability -Can capture complex relationships -High tolerance to noisy data -Ability to classify untrained patterns

Decision tree algorithms employ a ______ strategy, meaning that they grow the tree by making locally optimal decisions. Greedy Mutually exclusive Irreversible Randomized

Greedy

Good split attributes create _______ child nodes in decision trees. A large number of Heterogeneous Small size Homogeneous

Homogenous

Quality of Data: Duplicate Data

Major issue when merging data from heterogeneous sources- Data cleaning - > Process of dealing with duplicate data

General Structure of ANN

Multiple layers -Input layer (raw obs.) -Hidden layer(s) -Output layer Neurons (i.e. nodes) Weights (wij) Bias values (θj)

Why is the error rate on the training dataset not a good indicator of the error rate on future data?

No, because it is overly optimistic and also biased. Future data is itself different and new data so it would not be wise to use the previous data error rate as an indicator on future data.

Which of the following is not one of the reasons that decision trees are popular as a classification method? Black box pattern identification Light computational requirements Stable performance None of the above

None of the above

Methods of Estimation: Random Subsampling

Repeated holdout -In each iteration, a certain proportion is randomly selected for training -The error rates on the different iterations are averaged to yield an overall error rate

Visualization: Histogram

Shows the distribution of values of a single variable Divide the values into bins and show a bar plot of the number of objects in each bin The height of each bar indicates the number of objects Shape of histogram depends on the number of bins

Confusion Matrix

Separates out the decisions made by the classifier -Making explicit of how one class is being confused for another class

NN: User Inputs

Specify Network Architecture... Number of hidden layers •Most popular -one hidden layer Number of nodes in hidden layer(s) •More nodes capture complexity, but increase chances of overfitting Number of output nodes •For classification, one node per class (in binary case can also use one) •For numerical prediction use one Learning Rate •Low values "downweight" the new information from errors at each iteration •This slows learning, but reduces tendency to overfitto local structure

Recall:

TP / TP + FN a / a + b

TPR FPR

TPR = TP / TP +FN a /a + b FPR = FP / FP +TN c / c + d

The shape of a histogram for a continuous variable would depend on the number of bins used. T/F

TRUE

Accuracy of a classifier is given as: the percentage of the records that were correctly classified by the classifier The number of true positives given in the confusion matrix The number of true negatives given in the confusion matrix All of the above

The percentage of the records that were correctly classified by the classifier

Posterior probability is defined as: The probability of an event before observation The probability of an event after observation Unconditional probability of an event None of the above

The probability of an event after observation

In cross-validation, each record is used exactly once for testing purposes. T/F

True

Attribute:

a property or characteristic of an object-Examples: age of a person, eye color, etc.-[Attribute || Field || Variable || Feature]

The task of mapping an input attribute set into its class label is called Clustering Classification Segmentation Prediction

classification

NN: Used for.. Can capture.. Major Danger:

classification and prediction a very flexible/complicated relationship between the outcome and a set of predictors Overfitting

The performance of a model in terms of positive and negative frequencies can be depicted using a table known as the: Validation table Confusion matrix Validation matrix Sensitivity matrix

confusion matrix

________ is a method of dividing the existing data set into two parts in a way that each class label is represented with equal proportions in different parts. Segmentation Cross-validation Stratification Overfitting

cross - validation

Object

each row is one object-[Case || record || row || observation]-a set of measurements for one entity -e.g. the height, weight, age, etc. of one person

The steps involved in the classification process are: Model learning and model update Algorithm development and model learning Model learning and model application Algorithm development and model application

model learning and application

Conditional probability: : P(C|A1, A2, ..., An )

probability of an event after observation -P(C=Spam|exceeded=1, login=1, data=0, fsu=0) -Posterior, after evidence

Prior probability: P(C)

probability of an event before observation

Which of the following visualization tools provides information similar to correlation analysis?

scatter plot

Classification methods can be compared and evaluated according to their ______ speed, accuracy, scalability, robustness, interpretability

speed, accuracy, scalability, robustness, interpretability (all of the above)

NN Disadvantages

-Considered a "black box" prediction machine, with no insight into relationships between predictors and outcome -No variable-selection mechanism, so you have to exercise care in selecting variables -Heavy computational requirements if there are many variables (additional variables dramatically increase the number of weights to calculate)

_______ is a popular algorithm that uses artificial neural networks to learn from data. Backpropagation Apriori C5.0 Sequential covering

Backpropagation

Why are the assumptions of Naïve Bayes important? Note: E depicts the event and A depicts the observations Because they make the process more realistic Because they simplify the task of computing P(A|E) Because they simplify the task of computing P(A) All of the above

Because they simplify the task of computing P(A|E)

Categorical vs continuous

Categorical Attribute -Has only a finite or countable set of values -Examples: eye color, letter grade, zip code -Can be of different sub-types: nominal, ordinal and flag -Can be stored as string or numerical _______________________________ Continuous Attribute: - Quantitative attributes - Examples: weight, failure per hour, number of students, temperature -Can be of different sub-types: interval or ratio -Can be stored as numerical (real or integer)

Summary Stats: Central Tendency vs Spread

Central: Mean/Median, Frequency and Mode Spread: Percentiles, range and variance

Metrics for Performance Evaluation

Classification error -Classifying a record as belonging to one class when it belongs to another class -Count Error rate -Percent of misclassified records out of the total records

Which of the following is one of the tasks in the model learning step? Classify new / unknown records Estimate the accuracy of the model Develop the classification model All of the above

Develop the classification model

Issue w/ Naive Bayes: Continuous Attributes

Discretize the range into bins Two-way split:(A < v) or (A > v)•Choose only one of the two splits as new attribute Probability density estimation: •Assume attribute follows a normal distribution •Use data to estimate parameters of distribution (e.g., mean and standard deviation) •Once probability distribution is known, can use it to estimate the conditional probability P(Ai|c)

In this course we used the ...

Greedy strategy -Split the records based on an attribute test that optimizes certain criterion -in a divide and conquer manner

A prerequisite of the classification task is to: Introduce human intuition into the process Have predetermined class labels Use parallel computing All of the above

Have pre-determined class labels

Which of the following is not one of the disadvantages of artificial neural networks? Black box classification Long computation times Proneness to overfitting High tolerance to noisy data

High tolerance to noisy data

Some of the questions to ask after building the classification model are: How predictive is the model that was built? How can we obtain the prediction results? Given two or more models, which one is better? All of the above?

How predictive is the model that was built? How can we obtain the prediction results? Given two or more models, which one is better? (All of the above)

Existence of a large number of decision nodes and branches in a decision tree is an indication of: Random error Systematic error Entropy Overfitting

Overfitting

What is overfitting? Explain it in the context of training set model error rate and model complexity.

Overfitting according to our course PowerPoint is the tendency to tailor a model to the training data as much as possible. This occurs when the data model is confusing the noise in the training data with the signal. The noise referring occurrences due to chance and the signal referring to actual patterns within the data set. This can happen with a neural network model because it can overfit the data causing the error rate to be too large.

ROC Curve

ROC curve plots TP rate on the y-axis against FP rate on the x-axis at various cut-off settings Want a larger curve for a strong model

Quality of Data: Noise

Random error or variance in the original value -Examples: distortion of a person's voice when talking on a poor phone and "snow" on television screen

Methods of Estimation: Holdout

The holdout method reserves a certain amount for training and uses the remainder for testing -Commonly 70% for training and 30% for test For small datasets, samples might not be representative -Few or none instances of some classes -Stratified sampling: Balancing the data -Making sure that each class is represented with approximately equal proportions in both subsets (similar class distribution in each subset)

Some Questions after Building a Classification Model

•How predictive is the model that we built? •How reliable are the predicted results? •Which model is better (in terms of performance)?2

Which of the following is one of the stopping conditions that stop the growth of a decision tree? All records for a given node belong to the same class There are no more remaining attributes for further partitioning There are no more records left for splitting All of the Above

All records for a given node belong to the same class There are no more remaining attributes for further partitioning There are no more records left for splitting (All of the above)

Which of the following is not one of the user inputs required to run artificial neural network algorithms? Number of leaf nodes Learning rate # of hidden layers All of the above

Number of leaf nodes, Learning rate, # of hidden layers (All of the above)

Data Types

Record Data: consists of a collection of records, each of which consists of a fixed set of attributes Transaction Data: A special type of record data, where each record (transaction) involves a set of items Document Data: Each document becomes a 'word' vector •each word is a component (attribute) of the vector •the value of each component is the number of times the corresponding word occurs in the document Graph Data

Issue w/ Naive Bayes: The Zero-probability Problem

Solution: Use Laplacian correction -Add 1 to the count for every attribute value-class combination -Result: probabilities will never be zero P(Ai| Cj) = |Aij| + 1 / Nj+ v

Naive Bayes Definition

Statistical classifier -Perform probabilistic prediction -predict the probability of belonging to a class Combining pieces of evidence -Prior and posterior information

Choropleth Map

spatial data •Maps using color shadings to represent numerical values are called chloroplethmaps

Precision is computed as: The number of true positives divided by the total number of predicted positive records The number of true positives divided by the total number of actual positive records The number of true positives divided by the number of false positives The number of true positives divided by the number of true positives

The number of true positives divided by the total number of predicted positive records

Which of the following is one of the differences between a training data set and a test data set? Training data sets include class labels, whereas test data sets have unknown class labels Training data sets use old data, whereas test data sets use new data Training data sets are used to derive the classification model, whereas test data sets are used to estimate model performance All of the above

Training data sets are used to derive the classification model, whereas test data sets are used to estimate model performance

A decision tree is mutually exclusive, meaning that each data record is covered by at most one path in the tree. T/F?

True

A test set and a training set are similar in the sense that they both include observations with known class labels. T/F

True

Any classification model that has an area of greater than 0.5 under its ROC curve performs better than classification by chance. T/F

True

Entropy

a measure of the uncertainty associated with a random variable

Which of the following visualization tools can be used for outlier detection?

box plot

When a classification begins to memorize the training data than learning from a general trend, then ______ occurs. Random error Systematic error Overfitting Deviation

overfitting