ISDS 471 Exam 1
What do we usually observe if the model is overfitted?
Fitting training data very well Fitting validation data very badly
Interpretation of the beta coefficients
For some variable age, the beta coefficient is -150, means per unit increase in age, the selling price decrease by 150
Stepwise
Forward selection but at each step also may drop non-significant predictors like in backward elimination Drawback: R^2 are biased high, estimates for regression coefficients are biased high in absolute value, narrow confidence interval, s.e. of regression coefficient estimates are biased low
How many dummy variables can we create for a categorical variable with m levels?
M
What is homoscedasticity?
Variance of residuals are the same, check with scatter plot of residuals over fitted y All points have the same distance from the line
What is overfitting?
When we use the same data to build the model and assess performance Using validation to choose parameters Assessing multiple models on same validation
Which of the following is (are) assumption(s) for MLR? Normality Linearity Homoscedasticity All of the above
all of the above
Which of the following is unsupervised learning? - Data visualization - Data reduction - Cluster analysis - All of the above
all of the above
Why do we want to evaluate supervised learning tasks? Compare models Select the tuning parameter Learn the prediction or classification accuracy All of the above
all of the above
What is AUC
area under curve, most common metric 1, perfect discrimination between classes .5, no better than naive rule
Which of the following is not supervised learning? - Association rule - Classification - prediction
association rule
AE
average error, gives idea of systematic over or under prediction
For a data set with 200 observations and 200 variables, which of the following variable selection method do you think is inappropriate? -Backward -Best subset -Both
both
How do we check the assumption for normality? QQ plot Histogram Both
both
What type of data to use bar chart for
categorical
What can we use heatmap for?
check the missing value pattern with conditional formatting Check the collinearity
What is the naive rule?
classify all records as belonging to the most prevalent class Used as benchmark
How do we check the missing value pattern in excel?
conditional formatting --> new rule
What type of data to use histogram for
continuous
How do we check collinearity
correlation in data analysis toolbox
Which of the following is not a measure of prediction error? -error rate -average error -MAE -RMSE
error rate
What is ER
error rate accuracy = 1 - error rate
FNR
false negative rate # true C1 that are classified C0 / # true C1 % of C1 incorrectly classified
FPR
false positive rate # true C0 that are classified C1 / # true C0 % of C0 incorrectly classified
What are the 4 variable selection methods for MLR
forward, backward, stepwise, best subset (exhaustive search)
What is collinearity
high correlation among independent variables
What is collinearity?
high correlation among independent variables use pairwise correlation matrix to delete the variables when their correlation is above .8 or below -.8
MAD/MAE
mean absolute error (deviation), gives an idea of the magnitude of errors
MAPE
mean absolute percentage error
In an MLR model 1 with 12 predictors, you have adjusted R2 = 0.710 and Mallow's Cp = 13.09. In the other MLR model 2 with 13 predictors, you have adjusted R2 = 0.712 and Mallow's Cp = 11.35. Which model is better? -Model 1 -Model 2
model 1, mallow's cp is closer to the number of coefficients (13)
If data are partitioned into three parts, what is validation data not used for?
model valuation
What assumptions do we need in MLR?
normality, linearity, independence, homoscedasticity,
Why do we combine categories when there are too many categories for a categorical variable Avoid categories of too few observations - Reduce the number of dummy variables - Other two are correct
other two are correct
how many dummy variables do we need? how many dummy variables can we create?
p-1 p
What is ROC
receiver operating characteristic plots sensitivity on y-axis and 1-specificity on x-axis, positive relationship Curves closer to the top left = better performers Comparison curve is the diagonal, reflects performance of naive rule
RMSE
root-mean-squared-error (yhat - y)
What is linearity?
scatterplot of y over continuous x
if cut off increases, what happens to sensitivity, specificity, FPR, FNR
sensitivity decreases specificity increases FPR decreases FNR increases
Confusion matrix ........................Predicted class Actual ............1............ 0 1 ........................a.............b 0 ........................c............ d
summarizes correct and incorrect classifications from a dataset a and d give number of correct classifications b = number of class 1 incorrectly predicted as 0 c = number of class 0 incorrectly predicted as 1
Which dummy variable is usually considered as reference category?
the last category
If given a regression output, which variable should be deleted?
the one with a high p-value, usually higher than .05
What type of data to use line chart for
time series
SSE
total sum of squared error
Which data set do you conduct the model diagnosis on?
training data
What is independence?
when residuals or error terms are independent of each other, check with scatterplot of residuals over fitted y when points in a small chunk below or above regression line are grouped together
How many observations can XLminer handle for training data?
10,000 observations
What is the miclassification error rate? ..................Predicted class Actual ......1 ......2 ...........3 1 ................10........0 ..........2 2 ...............3 ........9............3 3.................1..........1..............11
10/40
What are partitioned data used for?
2 parts -Training (trains model) and validation (evaluates model) 3 parts -Training (trains model), validation (compares and selects model/parameters), test part (evaluates)
What is the sensitivity when cutoff is .6 Actual ......Prob(Y=1) 1....................93 1.....................91 1.....................76 0....................69 1.....................51 1.....................44 0.....................31 0.....................25 0.....................17 0.....................04
3/5 (sensitivity is # of C1 that are correctly classified as C1 / # true C1). We are saying everything above .6 is C1, that means we have three 1's that are correctly classified. But there are actually 5 1's in total.)
Example of the naive rule: Sample with 70% C'1 and 30% C0's, what is the error rate for the naive rule?
30% / .3 We are putting all observations in C1
The following 5 questions are based on the distribution chart below. Actual ......Prob(Y=1) 1....................93 1.....................91 1.....................76 0....................69 1.....................51 1.....................44 0.....................31 0.....................25 0.....................17 0.....................04 Using cutoff of .5, what is the value of a? ......................Predicted class Actual..........1.......0 1.....................a........b 0....................c.........d
4
What is a high correlation
> .8 or < -.8
Outliers
> Q3 + 1.5 * IQR < Q1 - 1.5 * IQR
Variance explained by the model
Adjusted R^2 select model with highest value, if 2 models have similar values, select the one whose mallow's cp is closest to the # of coefficients (variables + 1)
Best subset (exhaustive)
All possible subsets of predictors assessed, judge by adjusted R^2 Drawback: computationally intensive, not good for dimensional more than 23
Examples of supervised learning tasks
Classification (predict categorical target/outcome variable, binary) Prediction (predict numerical target/outcome variable)
How do we check the distribution of numeric variables in excel
Either plot box plot or histogram
FNR will increase if cutoff increases from .2 to .8 Actual ......Prob(Y=1) 1....................93 1.....................91 1.....................76 0....................69 1.....................51 1.....................44 0.....................31 0.....................25 0.....................17 0.....................04
# of true C1 that are classified as C0 / # of true C1 with .2, FNR = 0 with .8, FNR = 3/5 true
Specificity
# true C0 that are classified C0 / # true C0 % of C0 correctly classified
Sensitivity
# true C1 that are classified C1 / # true C1 % of C1 correctly classified
Which of the following is not a partial search algorithm? -Best subset selection -forward selection -backward selection -stepwise regression
-best subset selection
Which of the following variable selection method is best? -forward selection -backward selection -Stepwise regression -cannot be determined
-cannot be determined
What is the FPR when cutoff = .7 Actual ......Prob(Y=1) 1....................93 1.....................91 1.....................76 0....................69 1.....................51 1.....................44 0.....................31 0.....................25 0.....................17 0.....................04
0/5 (we are saying everything above .7 is 1. FPR is true C0 that are classified as C1 (which is nothing in our case) / # of true C0, which is five because there are five 0's below .7)
Using .5 as a cutoff, what is the value of c?
1 (everything above .5 is 1 for c, we are asking how many actual 0's are predicted as 1. There is one 1 above .5)
What is normality?
Assumes residuals are normally distributed, Histogram or residuals or QQ plot Take a slice of the data, should be dense in the middle and sparse on the sides Normall distributed around regression line
What is supervised learning?
Predict single target or outcome variable, often binary
Measures to evaluate prediction (how well does the model predict new data? Not how well it fits the data it was trained with)
RMSE, MAD/MAE, MAPE, AE, SSE
Measures to evaluate classification
ROC, AUC, ER, Sensitivity, Specificity, FPR, FNR, Confusion matrix
How to determine which variables are important?
Small p-values are important, if a p-value is greater than .05 it is not
Backward
Start with all predictors, eliminate least useful one by one, stop when all remaining predictors have statistically significant contribution Drawback: computing the initial model with all predictors can be time consuming and unstable, not good when n < p
Forward selection
Start with no predictors, add one by one starting with largest R^2, stop when addition no longer statistically significant Drawback: may miss pairs or groups of predictors that perform well together but perform poorly as single predictors
