Data Mining Chapter 7 - K-Nearest-Neighbor

Summary: For prediction, take average of the nearest neighbors. Choose appropriate k based on error rates/prediction metrics

"Curse of dimensionality" - need to limit # of predictors

Typical range of k:

1 - 20

If 3 NN of new record are 2 owners and 1 non-owner, what is the probability of a new household being an owner or non-owner?

2/3 chance of being owner, 1/3 for non-owner

What is K nearest neighbor?

Algorithm used for classification (of a categorical outcome) or prediction (of a numerical response)

What does k=1 mean?

Classify a new record as belonging to the same class as its nearest neighbor

What is the classification rule?

Compute distances between the validation records to be classified and existing records in the training set

KNN is ____, not model-driven

Data-driven

In prediction, what is usually used instead of the misclassification error rate to choose k?

RMSE or average error metric

What are the advantages of using KNN?

Simple and intuitive No assumptions about data Can be very powerful with a large training set

How do you measure 'nearby'?

The most popular distance measure is Euclidean distance between two records with predictor values

k.n.n. Makes no _____ about the data

assumptions

k is the number of nearby neighbors to be used to _____

classify the new record

When using weighted average, weight ______ with distance

decreases

Instead of relying on fitting a model like linear regression, KNN relies on ____

finding "similar" records in the training data

A drawback of using KNN is that the required size of training set ____ with # of predictors, p

increases exponentially

What is the Simplest case?

k=1, One-nearest neighbor.

Nearest neighbor means the same as

lowest Euclidean distance

For k > 5, Use a ____ decision rule to classify a record

majority

For cut-off value, Classification based on _____.

majority decision rule. "majority" is linked closely to cut-off value

High values of k provide more ___, less __, but may miss ____

more smoothing, less noise, but may miss local structure

To remove bias, put the _____ values in the formula

normalized

What does near mean?

records with similar predictor values X1, X2, ... Xp

How do you calculate the Euclidean distance?

square the differences of the record values, sum that, and then take the square root of the total

k=5 means use ____

the 5 nearest records

k = N is ___

the entire data set

Proportion of k neighbors that belong to a class is an estimate of ____

the probability of a new record belonging to that class

When using K-NN for Prediction, for Numerical Outcome, what's a better method than majority vote?

use average of response values from the k-nearest neighbors to get predicted value in the validation set

Low values of k capture _____ but also ____

local structure in data but also noise

Typically choose that value of k which has ____

lowest error rate in validation data

In KNN, Classify the record as whatever the _____ is among the nearby records, a.k.a the "neighbors"

predominant class

Majority decision rule says the record is classified as a member of ____

the majority class of the k neighbors

What is k.n.n. basic idea of categorical outcome?

For a given record to be classified, identify nearby records - each record consists of an outcome, or class membership, and predictor values

What is a drawback of using KNN with a large training set?

It takes a long time to find distances to all the neighbors and then identify the nearest ones.

Classification results can be reported as:

Misclassification error rate or Classification/confusion matrix

Summary: Find distance between record-to-be-classified and all other records in the training set

Select k-nearest records Classify it according to majority vote of nearest neighbors

When you have the entire data set, what rule do you use?

The "naïve rule"--classify all records according to majority class.

Sometimes predictors need to be standardized to ____ scales before computing distances

equalize