6414 FINAL MIX (15)
MSE=
Bias^2+Var
How will the procedure change if we test whether the coefficient is equal to a constant?
1) We reject the null hypothesis if the t-value is larger than t alpha over 2n-p-1, when the absolute value of t-value, is larger than the critical point. 2) OR we can look if the p-value is smaller than .01
what are three things that a regression analysis is used for?
1. Prediction of the response variable, 2. Modeling the relationship between the response and explanatory variables, 3. Testing hypotheses of association relationships
3 options for splitting the data
1. Random sampling 2. K-fold cross-validation 3. Leave-one-out cross-validation
How do we compute classification error? (2 ways)
1. Training error 2. Cross validation
Which one is correct? A. The prediction intervals need to be corrected for simultaneous inference when multiple predictions are made jointly. B. The prediction intervals are centered at the predicted value. C. The sampling distribution of the prediction of a new response is a t-distribution. D. All of the above.
3.2 - Knowledge Check 3
To correct for complexity for GLM, what can we use?
AIC and BIC
Outlier
Any data point that is far from the majority from the data in both x and y
conditional relationship
Capturing the association of a predicting variable to the response variable, conditional of other predicting variables in the model.
Conditional Relationship
Capturing the association oof a predicting variable to the response variable conditional of other predicting variables in the model
What do you add to prediction risk to get training risk?
Complexity penalty
L2 does not account for multicollenarity.
False
What is the rule of thumb for Cook's Distance?
If Di > 4/n or Di >1 or any "large" Di should be investigated
When do we reject the f-statistic? and what does this mean?
If it's larger than the critical point, with alpha being the significance level of the test. This means, again, that at least one of the coefficients is different from 0 at the alpha significance level.
In multiple linear regression, the model can be written in...?
Matrix form.
What method do we use to estimate the model parameters?
Maximum Likelihood Estimation approach
Do the estimated residuals have constant variance?
No
Mallows CP is an example of what?
Prediction risk
How do we interpret R^2?
Proportion of total variability in Y that can be explained by the regression (that uses X)
SST = ?
SSE + SSTR
Linearity assumption for a Logistic Model
Similar to the regression model we have learned in the previous lectures, the relationship we assume now, between the link, the g of the probability of success and the predicted variable, is a linear function.
In the case of complete separation, what should you do?
Simplify the model
T/F: The statistical inference for logistic regression relies on large size of the sample data.
T
Null-deviance
Test statistic for Overall Regression, shows how well the response variable is predicted by a model that includes only the intercept.
The fitted values are defined as
The regression line with parameters replaced with the estimated regression coefficients.
If the scatter plot of the residuals (epsilon ij) for the ANOVA is NOT random: (2)
The sample responses are not independent, or the variances of responses are not equal
The total sum of squares divided by N-1 is
The sample variance estimator assuming equal means and equal variances
The pooled variance estimator is:
The sample variance estimator assuming equal variances.
AIC uses estimated variance based on what?
The submodel
The objective of the pairwise comparison is
To identify the statistically significantly different means.
The L1 penalty results in lasso regression.
True
In the case of multiple linear regression, controlling variables are used to control for sample bias. (T/F)
True. See Lesson 3.4: Model Interpretation "Controlling variables can be used to control for bias selection in a sample."
The expectation of the mean response is:
UNBIASED
The estimators for the regression coefficients are:
Unbiased regardless of the distribution of the data.
predicting or explanatory (independent) variables
a set of other variables that might be useful in predicting or modeling the response variable (x1, x2)
MSSTr measures...
between-group variability
for our linear model where: Y = B0 + B1 + EPSILON (E), what does the epsilon represent?
deviance of the data from the linear model (error term)
The test of subset of coefficients tests the null hypothesis that
discarded variables have coefficients equal to zero.
An important aspect in prediction is....
how it performs in new settings.
B0 = ?
intercept
If we have a negative value for B1,....
is consistent with an inverse relationship between x and y
L2 penalty (ridge)...
is easy to implement, but it does not measure sparsity and does not perform variable selection
Models with many predictors have...
low bias, high variance
We'd like to have prediction with...
low uncertainty for new settings.
Goodness of fit
means that the model assumptions hold and fits the data well.
The sampling distribution of MLEs can be approximated by a...
normal distribution
We can make associated statements for...
observational studies
response (dependent) variables
one particular variable that we are interested in understanding or modeling (y)
What inference does 'testing a subset of coefficients' provide?
provides inferences on the predictive power of the model
B1 = ?
slope
If there is a coefficient that does NOT equal zero, what does that mean?
that at least one of these predictors included in the model has predictive power.
In the pairwise comparison, if the confidence interval only contains positive values, then we conclude...
that the difference in means in statistically positive
Rate parameter
the expectation of the response Yi, given the predicting variables, which is modeled as the exponential of the linear combination of the predicting variables since the link function between expectation and the predicting variables is the log function
The larger the number of variables is for training risk....
the larger the training risk
ANOVA is a linear regression model where...
the predicting factor is a (one) categorical variable.
Predictive factors
to best predict variability in the response regardless of their explanatory power
what is the objective of ANOVA?
to compare the means across the k populations, are they equal?
In a multiple linear regression model, the R^2 measures the proportion of total variability in the response variable that is captured by the regression model.
true
How will we diagnose the assumptions for ANOVA?
we are going to diagnose the assumptions on the residuals because under the error terms we do not know the means
How do we estimate prediction risk?
we can use an approach called Training Risk
What can we use to test if Betaj is = 0?
z test (wald test)
When the p-value of the slope estimate in the SLR is small the r-squared becomes smaller too.
False - When P value is small, the model fits become more significant and R squared become larger.
LOOCV is approximately AIC when the true variance is replaced by the estimate of the variance from the S submodel.
True
Lasso regression does not have a closed from expression for the estimated coefficients.
True
MLE is used for the GLMs for handling complicated link function modeling in the X-Y relationship.
True
The estimated regression coefficients from lasso regression as less efficient than those from the ordinary least squares estimation approach.
True
The first degree of freedom in the F distribution for any of the three procedures in stepwise is always equal to one.
True
The goal is to balance the trade-off between bias (more predictors) and variance (fewer predictors).
True
To get BIC in R, we use AIC with a k value of log(n).
True
Typically, confounding variables should be included in a model.
True
Variable selection is an art by itself.
True
We can combine ridge and lasso regression into what we call the elastic net regression.
True
When selecting variables, we need to first establish which variables are used for controlling bias selection in the sample and which are explanatory.
True
When the objective is of a model is prediction, correlated variables should be avoided.
True
When the objective is to explain the relationship to the response, one might consider including predicting variables which are correlated.
True
In multiple linear regression with idd and equal variance, the least squares estimation of regression coefficients are always unbiased.
True - the least squares estimates are BLUE (Best Linear Unbiased Estimates) in multiple linear regression.
Akaike Information Criterion (AIC)
A more general approach, for linear regression under normality this becomes training risk + penalty that looks like Mallow's EXCEPT the variance is the true variance not the estimate.
The following output was captured from the summary output of a simple linear regression model that relates the duration of an eruption with the waiting time since the previous eruption. Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -1.374016 A -1.70 0.045141 * waiting 0.043714 0.011098 B 0.000052 *** --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 0.4965 on 270 degrees of freedom Multiple R-squared: 0.8115, Adjusted R-squared: 0.8108 F-statistic: 1162 on 1 and 270 DF, p-value: < 2.2e-16 Using the table above, what is the standard error of the intercept, labeled A, and rounded to three decimal places? 2.336 0.808 0.806 -0.806 None of the above
0.808 See 1.4 Statistical Inference Std.Err = Estimate /t-value = -1.374016/-1.70 = 0.808
VIF =
1 / 1 - R^2
The objective of multiple linear regression is
1. To predict future new responses 2. To model the association of explanatory variables to a response variable accounting for controlling factors. 3. To test hypothesis using statistical inference on the model.
Objectives of Variable Selection
- High Dimensionality - Multicollinearity - Prediction vs Explanatory Objective
What are 4 reasons why the logistic model might not be a good fit?
1. there may be other variables that should be included in the model and/or the relationship between logit of the expected probability and predictors might be multiplicative rather than additive. 2. Initial observation outliers and leverage points are also still an issue for this model. The model should be fitted with and without these outliers. 3. the binomial distribution isn't appropriate (overdispersion) 4. the logit function does not fit well with the data. (there could be other s shaped functions that would work better)
Plotting the residuals versus fitted values checks for which assumption?
Constant variance & Independence
The 3 assumptions of ANOVA with respect to the error term:
Constant variance, independence, normality
3 ways Predicting Variables can be distinguished as:
Controlling, Explanatory, Predictive
What can we use to identify outliers?
Cook's Distance
How do we choose lambda?
Cross validation!
In Poisson regression: A) We make inference using t-intervals for the regression coefficients. B) Statistical inference relies on exact sampling distribution of the regression coefficients. C) Statistical inference is reliable for small sample data. D) None of the above.
D
The log-likelihood function is a linear function with a closed-form solution.
False Maximizing the log-likelihood function with respect to the coefficients in closed form expression is not possible because the log-likelihood function is non-linear.
If one confidence interval in the pairwise comparison does not include zero, we conclude that the two means are plausibly equal.
False
If the confidence interval for a regression coefficient contains the value zero, we interpret that the regression coefficient is definitely equal to zero.
False
If the non-constant variance assumption does not hold in multiple linear regression, we apply a transformation to the predicting variables.
False
If the p-value of the overall F-test is close to 0, we can conclude all the predicting variable coefficients are significantly nonzero.
False
If there are a group of correlated variables, lasso tends to select all of them.
False
If we do not reject the test of equal means, we conclude that means are definitely all equal
False
If we reject the test of equal means, we conclude that all treatment means are not equal.
False
In MLR, a VIFj of 10 means that there is no correlation among the jth predictor and the remaining predictors, here the variance of the estimated regression coefficient Bj is not inflated
False
In a multiple linear regression model with quantitative predictors, the coefficient corresponding to one predictor is interpreted as the estimated expected change in the response variable when there is a one unit change in that predictor.
False
In linear regression, outliers do not impact the estimation of the regression coefficients.
False
In ridge regression, when the penalty constant lambda equals to 1, the corresponding ridge coefficient estimates are the same as the ordinary least squares estimates
False
In the ANOVA, the number of degrees of freedom of the chi-squared distribution for the variance estimator is N-k-1 where k is the number of groups.
False
In the presence of near multicollinearity, the prediction will not be impacted.
False
In the presence of near multicollinearity, the regression coefficients will tend to be identified as statistically significant even if they are not.
False
In the regression model, the variable of interest for study is the predicting variable.
False
In the simple linear regression model, we lose three degrees of freedom because of the estimation of the three model parameters LaTeX: \beta_0,\:\beta_1,\sigma^2 β 0 , β 1 , σ 2 .
False
In the simple linear regression model, we lose three degrees of freedom because of the estimation of the three model parameters β 0 , β 1 , σ 2 .
False
Independence assumption can be assessed using the normal probability plot.
False
Independence assumption can be assessed using the residuals vs fitted values.
False
LaTeX: \beta_1 β 1 is an unbiased estimator for LaTeX: \beta_0 β 0 .
False
Observational studies allow us to make causal inference.
False
One-way ANOVA is a linear regression model with more than one qualitative predicting variables.
False
Only the log-transformation of the response variable can be used when the normality assumption does not hold.
False
Only the log-transformation of the response variable should be used when the normality assumption does not hold.
False
Prediction is the only objective of multiple linear regression.
False
Ridge regression does not have a close form expression for the estimated coefficients.
False
Suppose x1 was not found to be significant in the model specified with lm(y ~ x1 + x2 + x3). Then x1 will also not be significant in the model lm(y ~ x1 + x2).
False
T/F: Backward stepwise regression is preferable over forward stepwise regression because it starts with larger models.
False
T/F: Complex models with many predictors are often extremely biased, but have low variance.
False
T/F: Conducting t-tests on each β parameter in a multiple regression model is the best way for testing the overall significance of the model.
False
T/F: Given a qualitative predicting variable with 7 categories in a linear regression model with intercept, 7 dummy variables need to be included in the model.
False
T/F: If a predicting variable is categorical with 5 categories in a linear regression model with intercept, we will include 5 dummy variables in the model.
False
T/F: In regularized regression, the penalization is generally applied to all regression coefficients (β0, ... ,βp), where p = number of predictors.
False
T/F: In the case of a multiple linear regression model containing 6 quantitative predicting variables and an intercept, the number of parameters to estimate is 7.
False
T/F: In the case of a multiple regression model with 10 predictors, the error term variance estimator follows a χ 2 (chi-squared) distribution with n - 10 degrees of freedom.
False
T/F: It is good practice to create a multiple linear regression model using a linearly dependent set of predictor variables.
False
T/F: It is good practice to perform variable selection based on the statistical significance of the regression coefficients.
False
T/F: It is not required to standardize or rescale the predicting variables when performing regularized regression.
False
T/F: Mallow's Cp statistic penalizes for complexity of the model more than both leave-one-out CV and Bayesian information criterion (BIC).
False
T/F: Multiple linear regression captures the causation of a predicting variable to the response variable, conditional of other predicting variables in the model.
False
T/F: Predicting values of the response variable for values of the predictors that are within the data range is known as extrapolation.
False
T/F: Ridge regression is a regularized regression approach that can be used for variable selection.
False
T/F: Stepwise regression is a greedy algorithm searching through all possible combinations of the predicting variables to find the model with the best score.
False
T/F: The causation of a predicting variable to the response variable can be captured using multiple linear regression, conditional of other predicting variables in the model.
False
T/F: The equation to find the estimated variance of the error terms of a multiple linear regression model with intercept can be obtained by summing up the squared residuals and dividing that by n - p , where n is the sample size and p is the number of predictors.
False
T/F: The only objective of multiple linear regression is prediction.
False
T/F: The sampling distribution for estimating confidence intervals for the regression coefficients is a normal distribution.
False
T/F: The sampling distribution used for estimating confidence intervals for the regression coefficients is the normal distribution.
False
T/F: The training risk is an unbiased estimator of the prediction risk.
False
T/F: Variable selection is a simple and solved statistical problem since we can implement it using the R statistical software.
False
T/F: We can make causal inference in observational studies.
False
T/F: We can use the normal test to test whether a regression coefficient is equal to zero.
False
T/F: We interpret the coefficient corresponding to one predictor in a regression with multiple predictors as the estimated expected change in the response variable associated with one unit of change in the corresponding predicting variable.
False
T/F: When the number of predicting variables is large, both backward and forward stepwise regressions will always select the same set of variables.
False
The AIC is commonly used for prediction models since it penalizes the model complexity the most.
False
The F-test can be used to evaluate the relationship between two qualitative variables.
False
The causation effect of a predicting variable to the response variable can be captured using multiple linear regression, conditional of other predicting variables in the model.
False
The causation of a predicting variable to the response variable can be captured using Multiple linear regression, conditional of other predicting variables in the model.
False
The constant variance assumption is diagnosed by plotting the predicting variable vs. the response variable.
False
The constant variance is diagnosted using the quantile-quantile normal plot.
False
The estimated regression coefficient \beta^hat_i is interpreted as the change in the response variable associated with one unit of change in the i-th predicting variable .
False
The estimated regression coefficients will be the same under marginal and conditional model, only their interpretation is not.
False
The estimated variance of the error term has a \chi^2 distribution regardless of the distribution assumption of the error terms.
False
The estimator LaTeX: \hat \sigma^2 σ ^ 2 is a fixed variable.
False
The estimator σ ^ 2 is a fixed variable.
False
The interpretation of the regression coefficients is the same whether or not interaction terms are included in the model.
False
The larger then number of variables, the lower the training risk is.
False
The means of the k populations is a model parameter in ANOVA.
False
The number of degrees of freedom for the \chi^2 distribution of the estimated variance is n-p-1 for a model without intercept.
False
The number of degrees of freedom of the LaTeX: \chi^2 χ 2 (chi-square) distribution for the pooled variance estimator is LaTeX: N-k+1 N − k + 1 where LaTeX: k k is the number of samples.
False
The number of degrees of freedom of the χ 2 (chi-square) distribution for the variance estimator is N − k + 1 where k is the number of samples.
False
The only assumptions for a linear regression model are linearity, constant variance, and normality.
False
The only assumptions for a simple linear regression model are linearity, constant variance, and normality.
False
The penalty constant 𝜆 has the role to select the best subset of predicting variables.
False
The regression coefficient corresponding to one predictor is interpreted in a multiple regression in terms of the estimated expected change in the response variable when there is a change of one unit in the corresponding predicting variable.
False
The regression coefficient is used to measure the linear dependence between two variables.
False
The residuals have a t-distribution distribution if the error term is assumed to have a normal distribution.
False
The residuals have constant variance for the multiple linear regression model.
False
The residuals vs fitted can be used to assess the assumption of independence.
False
The sample means of the k populations is a model parameter in ANOVA.
False
The sample means of the k samples is a model parameter in ANOVA.
False
The sampling distribution for the variance estimator in ANOVA is LaTeX: \chi^2 χ 2 (chi-square) with N - k degrees of freedom.
False
The sampling distribution for the variance estimator in ANOVA is χ 2 (chi-square) regardless of the assumptions of the data.
False
The sampling distribution of the mean squared error is different of that of the estimated variance.
False
The statistical inference for linear regression under normality relies on large size of sample data.
False
The variables chosen for prediction and the variables chosen for explanatory objectives will be the same.
False
There are four assumptions needed for estimation with multiple linear regression: mean zero, constant variance, independence, and normality.
False
There are multiple model selection criteria that can be used and all provide the same penalization of the model complexity.
False
To get AIC in R, we use AIC with a value of k=1
False
Variable selection is solved statistical problem, particularly for a models with a large number of predictors.
False
We can test for a subset of regression coefficients only if we are interested whether additional explanatory variables should be considered in addition to the controlling variables.
False
We can test for a subset of regression coefficients to evaluate whether all regression coefficients corresponding to the predicting variables excluded from the reduced model are statistically significant.
False
We can test for a subset of regression coefficients using the F statistic test of the overall regression.
False
We cannot estimate a multiple linear regression model if the predicting variables are linearly independent.
False
We do not need to assume normality of the response variable for making inference on the regression coefficients.
False
the L2 penalty results in lasso regression.
False
β 1 is an unbiased estimator for β 0 .
False
Maximum Likelihood Estimation is not applicable for simple linear regression and multiple linear regression.
False - In SLR and MLR, the SLE and MLE are the same with normal idd data.
In interpretation of the regression coefficients is the same for logistic regression and Poisson regression
False - Interpretation of the regression coefficients of Poisson is in terms of log ratio of the rate. While in logistic regression it is in terms of log odd
In the GLMs the link function cannot be a non linear regression.
False - It can be linear, non linear, or parametric
It is a good practice to perform variable selection based on the statistical significance of the regression coefficients
False - It is not good practice to perform variable selection based on the statistical significance of the regression coefficients.
A Poisson regression model with p predictors and the intercept will have p + 2 parameters to estimate.
False - It'll have p + 1 parameter to estimate
Like standard linear regression we can use the F test to test for overall regression in logistic regression.
False - It's 1-pchisq(null deviance-residual deviance, DFnull-DFresidual)
Leave on out cross validation is preferred
False - K fold is preferred.
L2 penalty term measures sparsity
False - L1 penalty measures sparsity. L2 removes the limitation on variable selection
The L2 penalty measures the sparsity of a vector and forces regression coefficients to be zero
False - L1, Lasso, penalty measures sparsity of a vector and forces regression coefficients to be zero
The interpretation of the regression coefficients is the same for both Logistic and Poisson regressions.
False - Logistic : in terms of log odds. Poisson: in terms of log ratio of the rate.
The regression coefficients for the Poisson regression model can be estimated in exact/closed form.
False - MLE is NOT closed form.
If a Logistic Regression model provides accurate classification, then we can conclude that it is a good fit for the data
False - accuracy is not the same as good fit. GOF determine good fit. These two things don't have to co-exist.
It is good practice apply variable selection without understanding the problem at hand to reduce bias.
False - always understand the problem at hand to better select variables for the model.
When the number of predicting variables is large, both backward and forward stepwise regressions will always select the same set of variables
False - backward and forward regression will not always sslect the same set of variables
Backward stepwise regression is preferable over forward stepwise regression because it starts with larger models.
False - backward stepwise regression is more computational expensive than forward stepwise regression and generally selects a larger model.
We cannot use the training error rate as an estimate of the true error classification error rate because it is biased upward.
False - biased downward
For logistic regression we can define residuals for evaluating model goodness of fit for models with and without replication.
False - can only be with replication under the assumption that Yi is binary and n1 is greater than 1
It is good practice to perform a GOF test of Logistic Regression models without replications
False - can only do with replication on Logistic Regression
Complex models with many predictors are often extremely biased, but have low variance.
False - complex models with many predictors often have low bias but high variance
Trying all 3 link functions for a logistic regression (C-ln-ln, probit, logit) will produce models with the same GOF for a dateset
False - different link produce different output, thus the GOF would be different
In Logistics Regression, the estimated value for a regression coefficient Bi represents the estimated expected change in the response variable associated with one unit increase in the corresponding predicting variable xi holding the rest constant
False - expected change with respect to the odds of success
In MLR, the proportion of variability in the response variable that is explained by the predicting variables is called adjusted R2
False - explained by the model (not predicting variables)
In Logistic Regression, if the p-value of the deviance test for GOF is smaller than the significant level alpha, then it is plausible that the model is good fit
False - for GOF, if p - value is small meaning model is not a good fit (kind of reverse from significance of regression coefficients)
After fitting a logistic regression model, a plot of residuals versus fitted values is useful for checking if model assumptions are violated.
False - for logistic regression use deviance residuals.
The presence of multicollinearity in a MLR model will not impact the standard errors of the estimated regression coefficients.
False - it can impact the standard errors of the estimated regression coefficients.
Stepwise regression is a greedy algorithm searching through all possible combinations of the predicting variables to find the model with the best score
False - it does not guarantee the best score and not all possible combinations are checked
In Logistic regression, the error terms are assumed to follow a normal distribution
False - it doesn't have an error term
Least Square Elimination (LSE) cannot be applied to GLM models.
False - it is applicable but does not use data distribution information fully.
When a statistically insignificant variable is discarded from the model, there is little change in the other predictors statistical significance.
False - it is possible that when a predictor is discarded, the statistical significance of other variables will change.
For the testing procedure for subsets of coefficients, we compare the likelihood of a reduced model versus a full model. This is a goodness of fit test
False - it provides inference of the predictive power of the model
Forward stepwise will select larger models than backward.
False - it will typically select smaller models especially if p is large
The log-likelihood function is a linear function with a closed-form solution
False - log likelihood is non linear. A numerical algorithm is needed in order to maximize it.
Regression models are only appropriate for continuous response variables.
False - logistic and poisson model probability and rate
The logit link function is the best link function to model binary response data because it always fit the data better than other link functions.
False - logit function is not the only function that yields the s-shaped kind of curve.
Model with many predictors have high bias but low variance.
False - low bias and high variance
If the Cook's distance for any particular observation is greater than one, that data point is definitely a record error and thus needs to be discarded.
False - must see a comparison of data points. Is 1 too large?
If data on (Y, X) are available at only two values of X, then the model Y = \beta_1 X + \beta_2 X^2 + \epsilon provides a better fit than Y = \beta_0 + \beta_1 X + \epsilon.
False - nothing to determine of a quadratic model is necessary or required.
In a greenhouse experiment with several predictors, the response variable is the number of seeds that germinate out of 60 that are planted with different treatment combinations. A Poisson regression model is most appropriate for modeling this data
False - poisson regression models rate or count data.
If the confidence interval for a regression coefficient contains the value zero, we interpret that the regression coefficient is definitely equal to zero.
false
If the non-constant variance assumption does not hold in multiple linear regression, we apply a transformation to the predicting variables.
false
In ANOVA, the number of degrees of freedom of the chi-squared distribution for the variance estimator is N-k-1 where k is the number of groups.
false
In logistic regression, R^2 could be used as a measure of explained variation in the response variable.
false
In simple linear regression, the confidence interval of the response increases as the distance between the predictor value and the mean value of the predictors decreases.
false
T/F: Prediction is the only objective of multiple linear regression.
false
T/F: The proportion of variability in the response variable that is explained by the predicting variables is called correlation.
false
The F-test can be used to test for the overall regression in Poisson regression.
false
The interpretation of the regression coefficients is the same for both Logistic and Poisson regression.
false
The only assumptions for a simple linear regression model are linearity, constant variance, and normality.
false
We cannot estimate a multiple linear regression model if the predicting variables are linearly independent.
false
We do not need to assume independence between data points for making inference on the regression coefficients.
false
If a logistic regression model provides accurate classification, then we can conclude that it is a good fit for the data.
false "Goodness of fit doesn't guarantee good prediction." And conversely, good prediction doesn't guarantee the model is a good fit.
In a multiple linear regression model with n observations, all observations with Cook's distance greater than 4/n should always be discarded from the model.
false An observation should not be discarded just because it is found to be an outlier. We must investigate the nature of the outlier before deciding to discard it.
In a simple linear regression model, any outlier has a significant influence on the estimated slope parameter.
false An outlier does not necessarily have a large influence on model parameters. When it does, we call it an influential point.
Mallow's Cp statistic penalizes for complexity of the model more than both leave-one-out CV and Bayesian information criterion (BIC).
false BIC penalizes complexity more than the other approaches.
Backward stepwise regression is computationally preferable over forward stepwise regression.
false Backward stepwise regression is more computational expensive than forward stepwise regression and generally selects a larger model.
Complex models with many predictors are often extremely biased, but have low variance.
false Complex models with many predictors have often low bias but high variance.
You obtained a statistically significant F-statistic when testing for equal means across four groups. The number of unique pairwise comparisons that could be perfomed is seven
false For k=4 treatments, there are 𝑘(𝑘−1)/2 = 4(4−1)/2 = 6 unique pairs of treatments. The number of unique pairwise comparisons that could be perfomed is six.
Consider a multiple linear regression model with intercept. If two predicting variables are categorical and each variable has three categories, then we need to include five dummy variables in the model
false In a multiple linear regression model with intercept, if two predicting variables are categorical and both have k=3 categories, then we need to include 2*(k-1) = 2*(3-1) = 4 dummy variables in the model.
what are two ways to transform data?
power and log transformation
We do not interpret beta with respect to the response variable for a Poisson model but with....
respect to the ratio of the rate.
When conducting ANOVA, the larger the between-group variability is relative to the within-group variability, the larger the value of the F-statistic will tend to be.
true Given the formula of the F-statistic a larger increase in the numerator (between-group variability) compared to the denominator will result in a larger F-statistic ; hence, the larger MSSTr is relative to MSE, the larger the value of F-stat.
log rate
the log function of the expected value of the response
What is the interpretation of coefficient Beta in terms of logistic regression?
the log of the odds ratio for an increase of one unit in the predicting variable, holding all other variables constant
If one confidence interval in the pairwise comparison includes zero under ANOVA, we conclude that the two corresponding means are plausibly equal.
true
If there are specific variables that are required to control the bias selection in the model, they should be forced into the model and not be part of the variable selection process.
true
In ANOVA, to test the null hypothesis of equal means across groups, the variance of the response variable must be the same across all groups.
true
In a simple linear regression model, we can assess if the residuals are correlated by plotting them against fitted values.
true
In a simple linear regression model, we need the normality assumption to hold for deriving a reliable prediction interval for a new response.
true
In ridge regression, when the penalty constant lambda (λ) equals zero, the corresponding ridge coefficient estimates are the same as the ordinary least squares estimates.
true
Multicollinearity in multiple linear regression means that the columns in the design matrix are (nearly) linearly dependent.
true
Ridge regression can be used to deal with problems caused by high correlation among the predictors.
true
T/F: A partial F-test can be used to test whether a subset of regression coefficients are all equal to zero.
true
T/F: Analysis of variance (ANOVA) is a multiple regression model.
true
T/F: Before making statistical inference on regression coefficients, estimation of the variance of the error terms is necessary.
true
T/F: In the case of multiple linear regression, controlling variables are used to control for sample bias.
true
T/F: The estimated coefficients obtained by using the method of least squares are unbiased estimators of the true coefficients.
true
T/F: The regression coefficient corresponding to one predictor in multiple linear regression is interpreted in terms of the estimated expected change in the response variable when there is a change of one unit in the corresponding predicting variable holding all other predictors fixed.
true
The L1 penalty measures the sparsity of a vector and forces regression coefficients to be zero.
true
The estimators of the error term variance and of the regression coefficients are random variables.
true
The larger the coefficient of determination or R-squared, the higher the variability explained by the simple linear regression model.
true
The lasso regression requires a numerical algorithm to minimize the penalized sum of least squares.
true
The one-way ANOVA is a linear regression model with one qualitative predicting variable.
true
The penalty constant lambda (λ) in penalized regression controls the trade-off between lack of fit and model complexity.
true
We can assess the assumption of constant-variance in multiple linear regression by plotting the standardized residuals against fitted values.
true
We estimate the regression coefficients in Poisson regression using the maximum likelihood estimation approach.
true
A binary response variable with replications in logistic regression has a Binomial distribution.
true A binary response variable with replications does follow a Binomial distribution.
Suppose that we have a multiple linear regression model with k quantitative predictors, a qualitative predictor with l categories and an intercept. Consider the estimated variance of error terms based on n observations. The estimator should follow a chi-square distribution with n − k − l degrees of freedom.
true For this example, we use k + l df to estimate the following parameters: k regression coefficients associated to the k quantitative predictors, ( l − 1 ) regression coefficients associated to the ( l − 1 ) dummy variables and 1 regression coefficient associated to the intercept. This leaves n − k − l degrees of freedom for the estimation of the error variance.
Lambda
λ is a constant that balances the tradeoff between the lack of fit (measured by the SSE) and the complexity (measured by the penalty) which depends on the regression coefficients. The bigger λ, the bigger the penalty for model complexity
T/F: Cook's distance measures how much the fitted values (response) in the multiple linear regression model change when the ith observation is removed.
T
T/F: If the residuals are not normally distributed, then we can model instead the transformed response variable where the common transformation for normality is the Box-Cox transformation.
T
T/F: In Poisson regression, there is a linear relationship between the log rate and the predicting variables.
T
T/F: Influential points in multiple linear regression are outliers.
T
T/F: Multicollinearity can lead to less accurate statistical significance of some of the regression coefficients.
T
T/F: Multicollinearity in the predicting variables will impact the standard deviations of the estimated coefficients.
T
T/F: The assumptions to diagnose with a linear regression model are independence, linearity, constant variance, and normality.
T
T/F: The estimated regression coefficients in Poisson regression are approximate.
T
T/F: The estimator σ ^ 2 is a random variable.
T
T/F: The linear regression model under normality is also a generalized linear model with link function the identity link function.
T
T/F: The linear regression model with a qualitative predicting variable with k levels/classes will have k + 1 parameters to estimate
T
T/F: The logit function is the log of the ratio of the probability of success to the probability of failure. It is also known as the log odds function.
T
T/F: The logit link function is not the only S-shape function that can be used to model binary response data.
T
T/F: The mean sum of square errors in ANOVA measures variability within groups.
T
T/F: The prediction interval will never be smaller than the confidence interval for data points with identical predictor values.
T
T/F: The prediction of the response variable has higher uncertainty than the estimation of the mean response.
T
T/F: The presence of certain types of outliers can impact the statistical significance of some of the regression coefficients.
T
T/F: Under the normality assumption, the estimator for β 1 is a linear combination of normally distributed random variables.
T
T/F: We can use a Z-test to test for the statistical significance of a coefficient given all predicting variables in a Poisson regression model.
T
T/F: We can use a t-test to test for the statistical significance of a coefficient given all predicting variables in a multiple linear regression model.
T
T/F: We could diagnose the normality assumption using the normal probability plot.
T
T/F: When a Poisson regression does not fit well the data, it may that there may be more variability in the estimators than provided by the model.
T
T/F: When making a prediction for predicting variables on the "edge" of the space of predicting variables, then its uncertainty level is high.
T
T/F; For both logistic and Poisson regression, both the Pearson and deviance residuals should approximately follow the standard normal distribution if the model is a good fit for the data.
T
T/F; The estimator of the mean response is unbiased.
T
T/F: In logistic regression, there is not a linear relationship between the probability of success and the predicting variables.
T, the relationship is NOT linear but a linear FUNCTION.
A measure of the bias-variance tradeoff is the prediction risk
TRUE
Explanatory variable is one that explains changes in the response variable
TRUE
Mallow's CP is useful when there are no control variables.
TRUE
Bayesian Information Criterion
The complexity penalty is (# of predictors in submodel * true_variance of full model * log(n))/n BIC penalizes complexity more than other approaches and thus preferred in model selection for prediction BIC is similar to AIC except that the AIC complexity is further penalized by log(n)/2 For BIC, we need to specify k = log(n) Select the model with the smallest BIC
Characteristics of Spatial Regression
Trend: This can be long-distance increase/decrease in the data over space Periodicity/Seasonality: This is not common for spatial process Heteroskedasticity: This means the variability varies with space
If the constant variance assumption does not hold, we transform the response variable.
True
If the constant variance assumption in ANOVA does not hold, the inference on the equality of the means will not be reliable.
True
If the linearity assumption with respect to one or more predictors does not hold, then we use transformations of the corresponding predictors to improve on this assumption.
True
If the normality assumption does not hold, we transform the response variable, commonly using the Box-Cox transformation.
True
If the residuals of a MLR model are not normally distributed, we can model the transformed response variable instead, where a common transformation for normality is the Box-Cox transformation
True
If there are specific variables that are required to control the bias selection in the model, they should be force into the model and not be part of the variable selection process
True
If we reject the test of equal means, we conclude that some treatment means are not equal.
True
In Logistic Regression, we can perform residual analysis for binary data with replications
True
In Logistic regression, the relationship between the probability of success and predicting variables is non linear
True
In MLR, we can assess the assumption of Constant - Variance by plotting the standardized residuals against fitted valued
True
In Poisson Regression we do not interpret beta with respect to the response variable but with respect to the ratio of the rate.
True
In Poisson regression underlying assumption is that the response variable has a Poisson distribution, or responses could be wait times, or exponential distribution
True
In a Poisson regression model, we use a chi-squared test to test the overall regression.
True
Mean square error is commonly used in statistics to obtain estimators that may be biased, but less uncertain than unbiased ones. And that's preferred.
True
Multicolinearity in multiple linear regression means that the columns in the design matrix are (nearly) linearly dependent.
True
Multicollinearity can lead to misleading conclusions on the statistical significance of the regression coefficients of a MLR model
True
Multicollinearity in MLR means that the columns in the design matrix are (nearly) linearly dependent.
True
Multiple linear regression is a general model encompassing both ANOVA and simple linear regression.
True
Multiple linear regression is a general model encompassing both ANOVA and simple linear regression. correct
True
One problem with fitting a normal regression model to Poisson data is the departure from the assumption of constant variance
True
One reason why the logistic model may not fit is the relationship between logit of the expected probability and predictors might be multiplicative, rather than additive
True
Overdispersion is when the variability of the response variable is larger than estimated by the model
True
Partial F-Test can also be defined as the hypothesis test for the scenario where a subset of regression coefficients are all equal to zero.
True
Poisson distribution, the variance is equal to the expectation. Thus, the variance is not constant
True
Predicting variable is used in regression to predict the outcome of another variable.
True
Predictive power means that the predicting variables predict the data even if one or more of the assumptions do not hold.
True
Random sampling is computationally more expensive than the K-fold cross validation, with no clear advantage in terms of the accuracy of the estimation classification error rate.
True
Residual analysis can only be used to assess uncorrelated errors.
True
Ridge regression has a closed form expression for the estimated coefficients.
True
Simpson's Paradox - the reversal of association when looking at marginal vs conditional relationships
True
Stepwise regression is a greedy algorithm.
True
Studying the relationship between a single response variable and more than one predicting quantitative and/or qualitative variable is termed as Multiple linear regression.
True
T/F: Akaike Information Criterion (AIC) is an estimate for the prediction risk.
True
T/F: Controlling variables used in multiple linear regression are used to control for bias in the sample.
True
T/F: Elastic net regression uses both penalties of ridge and lasso regression and hence combines the benefits of both.
True
T/F: For a given predicting variable, the estimated coefficient of regression associated with it will likely be different in a model with other predicting variables or in the model with only the predicting variable alone.
True
T/F: If there are specific variables that are required to control the bias selection in the model, they should be forced into the model and not be part of the variable selection process.
True
T/F: In a multiple linear regression model with 6 predicting variables but without intercept, there are 7 parameters to estimate.
True
T/F: In multiple linear regression we study the relationship between a single response variable and several predicting quantitative and/or qualitative variables.
True
T/F: In multiple linear regression, we study the relationship between one response variable and both predicting quantitative and qualitative variables.
True
T/F: In order to make statistical inference on the regression coefficients, we need to estimate the variance of the error terms.
True
T/F: In ridge regression, when the penalty constant lambda (λ) equals zero, the corresponding ridge coefficient estimates are the same as the ordinary least squares estimates.
True
T/F: Ridge regression can be used to deal with problems caused by high correlation among the predictors.
True
T/F: The L1 penalty measures the sparsity of a vector and forces regression coefficients to be zero.
True
T/F: The error term in the multiple linear regression cannot be correlated.
True
T/F: The error term variance estimator has a (chi-squared) distribution with degrees of freedom for a multiple regression model with 10 predictors.
True
T/F: The estimated regression coefficient corresponding to a predicting variable will likely be different in the model with only one predicting variable alone versus in a model with multiple predicting variables.
True
T/F: The estimated regression coefficients are unbiased estimators.
True
T/F: The estimated variance of the error terms is the sum of squared residuals divided by the sample size minus the number of predictors minus one.
True
T/F: The hypothesis test for whether a subset of regression coefficients are all equal to zero is a partial F-test.
True
T/F: The lasso regression requires a numerical algorithm to minimize the penalized sum of least squares.
True
T/F: The penalty constant lambda (λ) in penalized regression controls the trade-off between lack of fit and model complexity.
True
T/F: We cannot estimate a multiple linear regression model if the predicting variables are linearly dependent.
True
T/F: We need to assume normality of the response variable for making inference on the regression coefficients.
True
T/F: When selecting variables for explanatory purpose, one might consider including predicting variables which are correlated if it would help answer your research hypothesis.
True
The ANOVA is a linear regression model with one or more qualitative predicting variables.
True
The ANOVA model with a qualitative predicting variable with LaTeX: k k levels/classes will have LaTeX: k+1 k + 1 parameters to estimate.
True
The L1 penalty results in a sparse matrix, and therefore does variable selection.
True
The L2 penalty does not result in a sparse matrix.
True
The LASSO regression requires a numerical algorithm to minimize the penalized sum of least squares
True
The LaTeX: R^2 R 2 value represents the percentage of variability in the response that can be explained by the linear regression on the predictors. Models with higher LaTeX: R^2 R 2 are always preferred over models with lower LaTeX: R^2 R 2 .
True
The Mallow's CP complexity penalty is two times the size of the model (the number of variables in the submodel) times the estimated variance divided by n.
True
The Partial F-Test can test whether a subset of regression coefficients are all equal to zero.
True
The Sum of Squares Regression (SSR) measures the explained variability captured by the regression model given the explanatory variables used in the model.
True
The advantage of having a biased model with less predicting variables is the reduction in uncertainty of predictions of future responses.
True
The decision in using ANOVA table for testing whether a model is significant depends on the normal distribution of the response variable
True
The deviance residuals are the signed square root of the log-likelihood evaluated at the saturated model
True
The equation to find the estimated variance of the error terms can be obtained by summing up the squared residuals and dividing that by n - p - 1, where n is the sample size and p is the number of predictors.
True
The estimated regression coefficient in lasso regression are obtained using a numeric algorithm.
True
The estimated regression coefficient in ridge regression are obtained using exact or closed form expression.
True
The estimated regression coefficients from Lasso are less efficient than those provided by the ordinary least squares
True
The estimators for the regression coefficients are Uunbiased regardless of the distribution of the data. correct
True
The estimators of the error term variance and of the regression coefficients are random variables.
True
The gam() function is a non-parametric test to determine what transformation is best.
True
The hypothesis testing procedures for subsets of regression coefficients is not used for GOF assessment in Logistic Regression
True
The l2 penalty results in ridge regression.
True
The larger K is, the larger the number of folds, the less bias the estimate of the classification the error is but has higher variability.
True
The larger the coefficient of determination or R-squared, the higher the variability explained by the simple linear regression model.
True
The lasso minimization problem is convex.
True
The least square estimation for the standard regression model is equivalent with Maximum Likelihood Estimation, under the assumption of normality.
True
The linear regression model with a qualitative predicting variable with k levels/classes will have k + 1 parameters to estimate
True
The log odds function, also called the logit function, which is the log of the ratio between the probability of a success and the probability of a failure
True
The logit function is the log of the ratio of the probability of success to the probability of failure. It is also known as log odds function
True
The mean sum of square errors in ANOVA measures variability within groups.
True
The number of parameters to estimate in the case of a multiple linear regression model containing 5 predicting variables and no intercept is 6.
True
The one-way ANOVA is a linear regression model with one qualitative predicting variable.
True
The penalty constant lambda in penalized regression control the trade-off between lack of fit and model complexity
True
The penalty constant 𝜆 has the role to control the trade-off between lack of fit and model complexity.
True
The prediction intervals are centered at the predicted value.
True
The prediction intervals need to be corrected for simultaneous inference when multiple predictions are made jointly.
True
The prediction of the response variable has higher uncertainty than the estimation of the mean response.
True
The prediction risk is the sum between the irreducible error and the mean square error
True
The regression coefficients can be estimated only if the predicting variables are not linearly dependent.
True
The regression coefficients that are estimated serve as unbiased estimators.
True
The sampling distribution of the estimated regression coefficients is Centered at the true regression parameters.
True
The sampling distribution of the estimated regression coefficients is centered at the true regression parameters.
True
The sampling distribution of the estimated regression coefficients is dependent on the design matrix.
True
The sampling distribution of the estimated regression coefficients is the t-distribution assuming that the variance of the error term is unknown an replaced by its estimate.
True
The sampling distribution of the prediction of a new response is a t-distribution.
True
The training risk is a biased estimator of the prediction risk
True
The variability of a submodel is smaller than the full model.
True
The variance of the response is equivalent to the expected value of the response in Poisson regression with no overdispersion.
True
To get AIC in R, we use AIC with a value of k=2
True
Under the normality assumption, the estimator for β 1 is a linear combination of normally distributed random variables.
True
Using leave-1-out Cross Validation is equivalent to K-fold to K-fold Cross Validation where the number of the folds is equal to the sample size of the training set.
True
Using stepwise regression, we can force variables into the model.
True
Variable selection can be used to deal with multicollinearity, reduce predictors and fit a model with more variables than observations.
True
We assess the assumption of constant-variance by plotting the residuals against fitted values.
True
We can assess the assumption of constant-variance in multiple linear regression by plotting the standardized residuals against fitted values.
True
We can do a partial F test to determine if variable selection is necessary.
True
We would like to have a prediction with low uncertainty for new settings. This means that we're willing to give up some bias to reduce the variability in the prediction.
True
When considering using generalized linear models, it's important to consider the impact of Simpson's paradox when interpreting relationship between explanatory variables and the response. This paradox refers to the reversal of these associations when looking at the marginal relationship compared to a conditional ones.
True
When estimating confidence values for the mean response for all instances of the predicting variables, we should use a critical point based on the F-distribution to correct for the simultaneous inference.
True
When selecting variables for a model, one needs also to consider the research hypothesis, as well as any potential confounding variables to control for
True
When the data may not be normally distributed, AIC is more appropriate for variable selection than adjusted R-squared
True
if VIF for each predicting variable is smaller than a certain threshold we can say that there is not problematic amount of multicollinearity in the MLR model
True
in MLR Regression, we could diagnose the normality assumption by using the normal probability plot
True
in MLR Regression, when using very large samples, replying on the p-values associated to the traditional hypothesis test with Ha: Bj not equal to 0 can lead to misleading conclusions on the statistical significance of the regression coeffcient
True
A negative value of β 1 is consistent with an inverse relationship between the predictor variable and the response variable. (T/F)
True See 1.2 Estimation Method "A negative value of β 1 is consistent with an inverse relationship"
L0 penalty, which is the number of nonzero regression coefficients
True - not feasible for a large number of predicting variables as requires fitting all models
Poisson Assumptions - log transformation of the rate is a linear combination of the predicting variables, the response variables are independently observed, the link function g is the log function
True - remember, NO ERROR TERM
Standard linear regression could be used to model Poisson regression using the variance stabilizing transformation sqrt(mu-3/8) if the number of counts is large
True - the number of counts can be small - then use Poisson
To estimate prediction risk we compute the prediction risk for the observed data and take the sum of squared differences between fitted values for sub model S and the observed values.
True - this is called training risk and it is a biased estimate of prediction risk
The g link function is also called the canonical link function.
True - which means that parameter estimates under logistic regression are fully efficient and tests on those parameters are better behaved for small samples.
We can use the z value to determine if a coefficient is equal to zero in logistic regression.
True - z value = (Beta-0)/(SE of Beta)
Where p the number of predictors is larger than n the number of observationsthe Lasso selects, at most, n variables
True when p is greater than n, lasso will select n variables at the most
The estimated regression coefficients in Poisson regression are approximate
True.
The pooled variance estimator, s pooled^2, in ANOVA is synonymous with the variance estimator, σ ^ 2, in simple linear regression because they both use mean squared error (MSE) for their calculations. (T/F)
True. See 1.2 Estimation Method for simple linear regression See 2.2 Estimation Method for ANOVA The pooled variance estimator is, in fact, the variance estimator.
A study was conducted to measure the effect of a fungicide treatment on the survival rate of botrytis blight. Botrytis blight samples were divided into 20 groups, each consisting of about 100 samples and exposed to different levels of chemicals in a fungicide. The output of a logistic regression model is below, where concS represents the concentration of a sulfur in the fungicide and concCu represents the concentration of a copper in the fungicide. Use it to answer the following multiple-choice questions. Call: glm(formula = cbind(Survived, Died) ~ concS + concCu,family = "binomial", data = data) Deviance Residuals: Min 1Q Median 3Q Max -9.5366 -2.4594 0.1223 3.9710 6.3566 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) 3.58770 0.22958 15.63 <2e-16 *** concS -4.32735 0.26518 16.32 <2e-16 *** concCu -0.27483 0.01784 15.40 <2e-16 *** Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 718.76 on 19 degrees of freedom Residual deviance: 299.43 on 17 degrees of freedom AIC: 363.53 What is the probability of survival for a botrytis blight sample exposed to a sulfer concentration of 0.7 and a copper concentration of 0.9? 0.826 0.674 0.311 0.577
0.577 exp(3.58770 - 4.32735*0.7 - 0.27483*0.9) / (1 + exp(3.58770 - 4.32735*0.7 - 0.27483*0.9)) = 0.577
A study was conducted to measure the effect of a fungicide treatment on the survival rate of botrytis blight. Botrytis blight samples were divided into 20 groups, each consisting of about 100 samples and exposed to different levels of chemicals in a fungicide. The output of a logistic regression model is below, where concS represents the concentration of a sulfur in the fungicide and concCu represents the concentration of a copper in the fungicide. Use it to answer the following multiple-choice questions. Call: glm(formula = cbind(Survived, Died) ~ concS + concCu,family = "binomial", data = data) Deviance Residuals: Min 1Q Median 3Q Max -9.5366 -2.4594 0.1223 3.9710 6.3566 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) 3.58770 0.22958 15.63 <2e-16 *** concS -4.32735 0.26518 16.32 <2e-16 *** concCu -0.27483 0.01784 15.40 <2e-16 *** Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 718.76 on 19 degrees of freedom Residual deviance: 299.43 on 17 degrees of freedom AIC: 363.53 The p-value for testing the overall regression can be obtained from which of the following? 1-pchisq(718.76,19) 1-pchisq(419.33,2) 1-pchisq(363.53,3) 1-pchisq(299.43,17
1-pchisq(419.33,2) The chi-squared test statistic is the difference between the null deviance (718.76) and the residual deviance (299.43), which is 419.33. The degrees of freedom is the difference between the null deviance degrees of freedom (19) and the residual deviance degrees of freedom (17), which is 2.
Akaike Information Criterion (AIC) is an estimate for the prediction risk
AIC indeed measures prediction risk.
When would we use the 'Comparing pairs of Means' method?
After we reject the null hypothesis of equal means
How can we diagnose multicollinearity?
An approach to diagnose collinearities through the computation of the Variance Inflation Factor, which you will compute for EACH predicting variable.
What is the hypothesis testing procedure for overall regression and what is it testing?
Analysis of Variance for multiple regression. We will use analysis of variance (ANOVA) to test the hypothesis that the regression coefficients are zero.
The estimated versus predicted regression line for a given x*: A. Have the same variance B. Have the same expectation C. Have the same variance and expectation D. None of the above
B. Have the same expectation 1.2 - Knowledge Check 3
Which one is correct? A. Independence assumption can be assessed using the residuals vs fitted values. B. Independence assumption can be assessed using the normal probability plot. C. Residual analysis can be used to assess uncorrelated errors. D. None of the above
C. Residual analysis can be used to assess uncorrelated errors. 1.3 - Knowledge Check 4
Classification Error Rate
Classification error rate is the probability that the new response is equal to the classifier(R) R is between 0 and 1. Most common value for R is 0.5 however a different R can be used to improve the prediction accuracy
Classification
Classification is prediction of binary responses. If the predicted probability is large, then classify y star as a success
Logistic Regression
Commonly used for modeling binary response data. The response variable is a binary variable, and thus, not normally distributed. In logistic regression, we model the probability of a success, not the response variable. In this model, we do not have an error term
A variable that is related to both a predictor and a response. For example outdoor temperature as it relates to ice cream sales and home invasions.
Confounding variable
Constant Variance Assumption
Constant variance assumption means that it cannot be true that the model is more accurate for some parts of the population, and less accurate for other parts A violation of this assumption means that estimates are not as efficient as they could be in estimating the true parameters. It also results in poorly calibrated prediction intervals
Used to control for bias selection in a sample.
Controlling variable.
How else can we estimate the classification error without the need of observing new data?
Cross validation
The estimators for the regression coefficients are: A. Biased but with small variance B. Biased with large variance C. Unbiased under normality assumptions but biased otherwise. D. Unbiased regardless of the distribution of the data.
D. Unbiased regardless of the distribution of the data. 1.2 - Knowledge Check 2
The estimators for the regression coefficients are: A. Biased but with small variance B. Unbiased under normality assumptions but biased otherwise. C. Biased regardless of the distribution of the data. D. Unbiased regardless of the distribution of the data.
D. Unbiased regardless of the distribution of the data. 3.2 - Knowledge Check 2
Leverage points
Data points that are far from the mean of the x's
leverage points
Data points that are far from the mean of the x's
Data w/ replications vs Data w/o replications
Data with replications: We can observe binary data for repeated trials. That is a binomial distribution with more than one trial or ni greater than 1 Data without replications: For each unique set of the observed predicting variables, we can observe binary data with no repeated trials. That is a binomial distribution with one trial where ni = 1
A data point far from the mean of the x's and y's is always: A. an influential point and an outlier B. a leverage point but not an outlier C. an outlier and a leverage point D. an outlier but not a leverage point E. None of the above
E. None of the Above. See 1.9 Outliers and Model Evaluation We only know that the data point is far from the mean of x's and y's. It only fits the definition of a leverage point because the only information we know is that it is far from the mean of the x's. So you can eliminate the answers that do not include a leverage point. That leaves us with remaining possibilities, "a leverage point but not an outlier" and "an outlier and a leverage point" , both of which we can eliminate. We do not have enough information to know if it is or is not an outlier . None of the answers above fit the criteria of it being always being a leverage point.
T/F: A goodness-of-fit test should always be conducted after fitting a logistic regression model without repetition.
F
T/F: For assessing the normality assumption of the ANOVA model, we can only use the quantile-quantile normal plot of the residuals.
F
T/F: If a logistic regression model provides accurate classification, then we can conclude that it is a good fit for the data.
F
T/F: If one confidence interval in the pairwise comparison does not include zero, we conclude that the two means are plausibly equal.
F
T/F: If the VIF for each predicting variable is smaller than a certain threshold, then we can say that multicollinearity does not exist in this model.
F
T/F: If the non-constant variance assumption does not hold in multiple linear regression, we apply a transformation to the predicting variables.
F
T/F: Other link functions for the Poisson regression model are c-log-log and probit.
F
A multiple linear regression model with p predicting variables but no intercept has p model parameters.
False
BIC penalizes complexity less than other approaches.
False
Backward stepwise regression adds one variable to the model at a time starting with the full model.
False
Backward stepwise regression can be performed if p>n.
False
Backward stepwise regression is preferable over forward stepwise regressioin.
False
For testing if a regression coefficient is zero, the normal test can be used.
False
In the presence of near multicollinearity, the coefficient of variation decreases.
False
T/F: Observational studies allow us to make causal inference.
False
The Box-Cox transformation is commonly used to improve upon the linearity assumption.
False
Under Logistics Regression, the sampling distribution used for a coefficient estimator is a Chi-squared distribution when the sample size is large.
False - sampling (or coef estimator) follos an approximately Normal
The slope of a linear regression equation is an example of a correlation coefficient.
False - the correlation coefficient is the r value. Will have the same + or - sign as the slope.
Observational Studies
For observation studies, causality is rarely implied
When do we use transformations?
If the linearity assumption with respect to one or more predictors does not hold, then we use transformations of the corresponding predictors to improve on this assumption. If the normality assumption does not hold, we transform the response variable, commonly using the Box-Cox transformation. If the constant variance assumption does not hold, we transform the response variable.
What does it mean when we reject the H0 of the F test?
If we reject the null hypothesis, we will conclude that at least one of the predicting variables has explanatory power for the variability in the response.
OLS vs Robust Regression
In OLS, we minimize the sum of square root errors to estimate the expectation of Y, given the predictor variables In robust regression, minimize the sum of absolute errors to estimate the median of Y, given the predictor variables Both the expectation and the median are measures of centrality of the distribution. However, median is robust to outliers, whereas the mean or the expectation is not The estimated confidence intervals are wider for robust regression than for ordinary least squares. And thus, the statistical inference is more conservative
Regularized Regression (Variable Selection)
In regularized regression for variable selection, we need to first re-scale all the predicting variables in order to be comparable on the same scale.
What is the difference between using Poisson regression versus the standard regression with, say, with the log transformation of the response variable?
In standard regression, the variance is assumed constant. In Poisson, the variance of the response is assumed to be equal to the expectation, since for the Poisson distribution, the variance is equal to the expectation. Thus the variance is not constant.
Stepwise regression is a greed algorithm, what does that mean?
It does not guarantee to find the model with the best score.
How we interpret ^B0?
It is the estimated expected value of the response variable, when the predicting variable equals zero.
Ridge Regression
It is used to correct for the impact multicollinearity This is commonly used to fit a model under multicollinearity; not used for model selection, does not force any betas to 0 For this model, the penalty is the sum of square regression coefficients times the lambda constant Minimizing the penalty provides a closed-form expression for the estimated regression coefficients When λ = 0; low bias, high variance λ = 1; betas = 0, high bias, low variance We have a closed-form expression for the estimated regression coefficients under this model Some of the regression coefficients may intersect the 0 line for large effective degrees of freedom but for others, the regression coefficients increase Alpha = 0
If our ANOVA model has an intercept, then how many dummy variables and why?
K-1 because of linear dependence between the X's
In regularized regression, what balances the bias variance tradeoff?
Lambda
Plotting the residuals versus each predictor checks for which assumption?
Linearity
Linearity Assumption - Poisson
Linearity can be evaluated by plotting the log of the event rate versus the predicting variables We can also evaluate linearity on the assumption of uncorrelated responses using the scatter plot for the residuals versus the predicting variables
What are the 4 assumptions of MLR?
Linearity, Constant Variance, Independence, Normality
Variable Selection Approaches
Mallows Cp AIC BIC Cross Validation
Parameter estimation
Maximizing the log likelihood function with respect to beta0, beta1 etc in closed (exact) form expression is not possible because the log likelihood function is a non-linear function in the model parameters i.e. we cannot derive the estimated regression coefficients in an exact form Use numerical algorithm to estimate betas (maximize the log likelihood function). The estimated parameters and their standard errors are approximate estimates
If we reject the null hypothesis for overall regression, what does that mean
Meaning that the overall regression has statistically significant power in explaining the response variable.
Constant variance assumption
Means that it cannot be true that the model is more accurate for some parts of the population, and less accurate for other parts of the populations. This can result in less accurate parameters and poorly-calibrated prediction intervals.
Using MLE, can we derive estimated coefficients/parameters in exact form?
No, they are approximate estimated parameters
What does the QQ plot and histogram check for?
Normality
Regression Analysis
Overview of the following models: - Linear Regression - ANOVA Regression - Multiple Linear Regression - Logistic Regression - Poisson Regression
Pearson Residuals - Poisson Regression
Pearson residuals is the standardized difference between the ith observed response, and estimated expected rate of events lambda i hat divided by the square root of the variance where the variance is equal to lambda i hat We need to standardized the difference between observed and expected response since their responses have different variances Normal Distribution
Biased Regression Penalties
Penalize large values of betas jointly. This should lead to multivariate shrinkage of the vector of regression coefficients
What are three ways we can transform the predicting variables?
Power, Log, Polynomial transformations
ANOVA
Relationship between the response variable y and one or more qualitative/categorical variables We can write the response as a sum between the mean of the category from which the response is observed plus an error term epsilon Assumptions: Same as linear regression, except that we do not have the linearity assumption, since we're not considering a relationship with a quantitative variable The model parameters are the group mean parameters, along with the variance of the error terms. The mean parameters are estimated as the group sample means
Logistic Regression with replications
Residuals: We can only define residuals for binary data with replications Goodness of Fit: We perform goodness of fit only for logistic regression with replications under the assumption that Yi is binomial with ni greater than 1
what is the relationship between s^2 and sigma^2?
S^2 estimates sigma^2
Elastic Net Regression
Simultaneously performs variable selection and continues shrinkage and can select groups of correlated variables. Elastic net often out performs the lasso in terms of prediction accuracy The L1 penalty generates a sparse model that enforces some of the regression coefficients to be 0. The L2 penalty removes the limitation of the number of selected variables, encourages group effect, stabilizes the L1 regularization path Alpha = 0.5 (between 0 & 1)
Elastic Net
Simultaneously performs variable selection and continues shrinkage and can select groups of correlated variables. Elastic net often out performs the lasso in terms of prediction accuracy.
HW 4 After this
Space
Spatial Process Assumptions
Stationarity: This means that the probability distribution of the spatial process does not change when shifted in space Isotropic: This means that the distribution of Ys is the same in all orientations
How can we find that a normality transformation of the predictive variable will improve the fit?
Such transformation can be identified by comparing the logit of the success rate, versus the predicted variables.
T/F: A negative value of β 1 is consistent with an inverse relationship between x and y.
T
T/F: Cross-validation using random sampling is less computationally efficient (more computationally expensive) in estimating the model error rate than the K-fold cross validation.
T
T/F: For a classification model, training error tends to underestimate the true classification error rate of the model.
T
T/F: For logistic regression, if the p-value of the deviance test for goodness-of-fit is large, then it is an indication that the model is a good fit.
T
T/F: If one confidence interval in the pairwise comparison includes only positive values, we conclude that the difference in means is statistically significantly positive.
T
T/F: If the constant variance assumption in ANOVA does not hold, the inference on the equality of the means will not be reliable.
T
T/F: We can perform goodness-of-fit analysis for a Poisson regression.
T
Variable selection addresses multicolinearity, high dimensionaltiy, and prediction vs explanatory prediction
TRUE
Variable selection is not special, it is affected by highly correlated variables
TRUE
Statistical Inference for Logistic Regression is not reliable for small sample size
TRUE - Only Large sample size
The R-squared and adjusted R-squared are not appropriate model comparisons for non linear regression but are for linear regression models.
TRUE - The underlying assumption of R-squared calculations is that you are fitting a linear model.
If p is larger than n, stepwise is feasible
TRUE - for forward, but not backward
The deviance and pearson residuals are normally distributed
TRUE - the residual deviances are chi square distributed
Stepwise is a heuristic search
TRUE it is also a greedy search that does not guarantee to find the best score
Linearity Assumption
The Logit transformation of the probability of success is a linear combination of the predicting variables. The relationship may not be linear, however, and transformation may improve the fit The linearity assumption can be evaluated by plotting the logit of the success rate versus the predicting variables. If there's a curvature or some non-linear pattern, it may be an indication that the lack of fit may be due to the non-linearity with respect to some of the predicting variables
Assumptions
The assumption that the errors are normally distributed is needed if we want to do any confidence or prediction intervals or hypothesis tests If this assumption is violated, hypothesis test and confidence and prediction intervals can be very misleading
Regression Coefficients
The estimated regression coefficients still have a closed form expression. However, we correct for the variance-covariance matrix of the error terms. The estimated regression coefficients remain unbiased, but the variance of the coefficients changes. The sample distribution of beta estimators also remains to be normal under the normality assumption
Statistical Inference for Poisson Regression
The estimators for the regression coefficients in the Poisson regression are approximately unbiased. The mean of the approximate normal distribution is beta. This approximation relies on the large sample data
What is the f-statistic?
The f statistic is going to be the ratio between the mean sum of square regression and mean sum of square of error.
Goodness of Fit Test
The null hypothesis is that the model fits well. The alternative is that the model does not fit well The test statistic for the goodness of fit test is the sum of squared deviances which has a Chi-Square distribution with n- p- 1 degrees of freedom If the p-value is small, we reject the null hypothesis of good fit, and thus we conclude that the model is not a good fit. We want LARGE values of P. Large values of P indicate that the model may be a good fit For goodness of fit test, we compare the likelihoods of the saturated model versus the fitted model.
L0 penalty
The number of nonzero regression coefficients. This is equivalent to searching over all models and not viable for a large p.
Classification
The prediction of binary responses. Classification is nothing more than a prediction of the class of your response, y* (y star), given the predictor variable, x* (x star). If the predicted probability is large, then classify y* as a success.
The fitted values are defined as:
The regression line with parameters replaced with the estimated regression coefficients.
Response variable
The response data are Bernoulli or binomial with one trial with probability of success
MLE
The resulting log-likelihood function to be maximized, is very complicated and it is non-linear in the regression coefficients beta 0, beta 1, and beta p MLE has good statistical properties under the assumption of a large sample size i.e. large N For large N, the sampling distribution of MLEs can be approximated by a normal distribution The least square estimation for the standard regression model is equivalent with MLE, under the assumption of normality. MLE is the most applied estimation approach
Binomial Data
This is binary data with repititions
Log Rate
This is the log function of the expected value of the response ln(λ(x)) = β0 + β1x
The objective of the residual analysis is
To evaluate departures from the model assumptions
Generalized Linear Model
To generalize the standard regression model to response data that do not have a normal distribution, this generalizes the linear model to response data coming from other distributions.
A negative value of β 1 is consistent with an inverse relationship between x and y .
True
A no-intercept model with one qualitative predicting variable with 3 levels will use 3 dummy variables.
True
In a multiple regression problem, a quantitative input variable x is replaced by x − mean(x). The R-squared for the fitted model will be the same
True
In a simple linear regression model, the variable of interest is the response variable.
True
In all the regression models we have considered (including multiple linear, logistic, and Poisson), the response variable is assumed to have a distribution from the exponential family of distribution.
True
Logistic Regression deals with the case where the dependent variable is binary, the conditional distribution is Binomial
True
Under the normality assumption, the estimator for LaTeX: \beta_1 β 1 is a linear combination of normally distributed random variables.
True
A Poisson regression model fit to a dataset with a small sample size will have a hypothesis testing procedure with more type I errors than expected
True -
The mean sum of squared errors in ANOVA measures variability within groups. (T/F)
True. See 2.4 Test for Equal Means. MSE = within-group variability
Cook's distance (Di) measures how much the fitted values in a multiple linear regression model change when the ith observation is removed. (T/F)
True. See Lesson 3.11: Assumptions and Diagnostics "This is the distance between the fitted values of the model with all the observations versus the fitted values of the model discarding the i-th observation from the data used to fit the model. "
The presence of certain types of outliers, such as influential points, can impact the statistical significance of some of the regression coefficients. (T/F)
True. See Lesson 3.11: Assumptions and diagnostics Outliers that are influential can impact the statistical significance of the beta parameters.
An example of a multiple linear regression model is Analysis of Variance (ANOVA). (T/F)
True. See Lesson 3.2 Basic Concepts "Earlier, we contrasted the simple linear regression model with the ANOVA model... Multiple linear regression is a generalization of both models."
Curse of Dimensionality
Using non-parametric regression with an increasing number of predicting variables p, there are many, many possible regression functions F To maintain a given degree of accuracy of an estimator, the sample size must increase exponentially with the dimension p
If p is large, what can we do instead?
We can perform a heuristic search, like stepwise regression
Backward Stepwise Regression
We start with all predictors in the full model and drop one predictor at a time Backward stepwise regression cannot be performed if p > n This is also more computationally expensive than forward stepwise regression This selects larger models if p is large
Estimation of WLS
We use a least squares approach but we need to account for the variance-covariance matrix sigma in weighting the error terms by their standard errors When there is correlation (sigma is not diagonal) we need to decorrelate the errors. Thus, we'll add the inverse of the sigma matrix to the least squares objective function
What can we do if the Normality or Constant Variance assumption does not hold?
We would transform the response variable. A common transformation is the Box-Cox transformation.
Prediction vs Explanatory Objective
When the objective is to explanatory, include predicting variables which are correlated For prediction, this should be avoided.
Multicollinearity
When the predicting variables are correlated, it is important to select variables in such a way that the impact of multicollinearity is minimized
When is Mallow's Cp useful?
When there are no control variables
Regularized Regression
Without Penalization: Estimate betas by minimizing the SSE With Penalization: Estimate betas by minimizing the penalized SSE λ*Penalty The bigger λ, the bigger the penalty for model complexity
Do the error terms have constant variance?
Yes
Will R^2 always increase when we add predicting variables?
Yes
Is the predicted regression line is the same as the estimated regression line at x*? How does it affect confidence intervals?
Yes, but the prediction confidence interval is wider than the estimation confidence interval because of the higher variability in the prediction.
Is training risk biased? why or why not?
Yes, the training risk is a biased estimate of prediction risk because we use the data twice. Once for fitting the model S and once for estimating the prediction risk. Thus, training risk is biased upward.
What is the distribution of binary data WITHOUT replications?
a binomial distribution with one trial where ni = 1
lambda
a constant that has the role of balancing the tradeoff between lack of fit and model complexity
What is B^ in MLR?
a linear combination of Y's and is normally distributed.
design matrix
a matrix consisting of columns of predicting variables including the column of ones corresponding to the intercept:
prediction risk
a measure of the bias-variance tradeoff
linear relationship
a simple deterministic relationship between 2 factors, x and y
correlation coefficient
a statistic that efficiently summarizes how well the X's are linearly related to Y
R^2 or coefficient of determination
a statistic that efficiently summarizes how well the x can be used to predict the response variable.
A linear regression model was fitted to estimate the response variable Height for black cherry trees using just the Diameter. The data frame has 31 observations. Here is the model summary, with some parts missing. Coefficients: Estimate Std. Error t-value Pr(>|t|) (Intercept) 62.0313 A 14.152 1.49e-14 *** Diameter 1.0544 0.3222 3.272 0.00276 ** --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 5.538 on B degrees of freedom Multiple R-squared: 0.2697, Adjusted R-squared: 0.2445 What is the MSE for this model? a. 30.669 b. 11.201 c. 20.534 d. None of the above
a. 30.669 MSE is the square of the Residual standard error. MSE = 5.5382 = 30.669
Adjusted R^2
adjusted for the number of predictive variables. So it's not going to increase as we add more predictive variables
If our ANOVA model does not have an intercept, then how many dummy variables?
all k dummy variables
3 primary objectives of ANOVA
analyze the variability of data, test whether the means are equal, estimate confidence intervals
Pearson Residuals
as the standardized difference between the ith observed response and estimated expected response, which is ni times the probability of success.
A linear regression model was fitted to estimate the response variable Height for black cherry trees using just the Diameter. The data frame has 31 observations. Here is the model summary, with some parts missing. Coefficients: Estimate Std. Error t-value Pr(>|t|) (Intercept) 62.0313 A 14.152 1.49e-14 *** Diameter 1.0544 0.3222 3.272 0.00276 ** --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 5.538 on B degrees of freedom Multiple R-squared: 0.2697, Adjusted R-squared: 0.2445 What is the value of the correlation coefficient between Height and Diameter? a. 0.2697 b. 0.5193 c. 0.3222 d. None of the above
b. 0.5193 In simple linear regression, the correlation coefficient between the response and predictor variables, 𝜌 = √𝑅2. Since, 𝑅 2 = 0.2697, then 𝜌 = √0.2697 = 0.5193.
conditional model (MLR)
captures the association of a predictor variable to the response variable, conditional of other predicting variables in the model.
Trying all three link functions for a logistic regression model (C-ln-ln, probit, logit) will produce models with the same goodness of fit for a dataset.
false
For Generalized Linear Models, including Poisson regression, the deviance residuals should approximately follow the standard normal distribution if the model is a good fit for the data.
false The deviance residuals are approximately N(0,1) if the model is a good fit.
If p is small,....
fit all submodels
what does residual analysis NOT check for? (for SLR assumptions)
independence
To test if a coefficient is less than a critical value, C, we conduct a one-sided test on the _________ tail of a ___________ distribution. left, normal left, t right, normal right, t None of the above
left, t See 1.4 Statistical Inference "For β 1 greater than zero we're interested on the right tail of the distribution of the β ^ 1."
ANOVA
linear regression with one or more qualitative predicting variables
simple linear regression
linear regression with one quantitative predicting variable
g link function
link the probability of success to the predicting variables
Forward-Backward stepwise regression
meaning adding and discarding one variable at a time iteratively.
Predictive power
means that the predicting variables predict the data even if one or more of the assumptions do not hold.
Multiple linear regression
multiple quantitative and qualitative predicting variables
The F-test is a _________ tailed test with ______ and ______ degrees of freedom. one, k, N-1 one, k-1, N-k two, k-1, N-k two, k, N-1 None of the above.
one, k-1, N-k See 2.4 Test for Equal Means The F-test is a one tailed test that has two degrees of freedom, namely k − 1 and N − k.
What is 'X*'?
predictor
In logistic regression, we model the__________________, not the response variable, given the predicting variables.
probability of a success
What inference does 'saturated vs fitted model' provide?
provides inferences on the goodness of the model
What kind of variable is a response variable and why?
random, because it varies with changes in the predictor/s along with other random changes.
For Poisson regression, the variance = ?
rate lambda
F-test measures...
ratio of between-group variability and within-group variability
Simpson's paradox
refers to reversal of an association when looking at a marginal relationship versus a partial or conditional one. This is a situation where the marginal relationship has a wrong sign.
Do we evaluate normality using residuals or the response variable?
residuals
What is the sampling distribution of ^B1?
t distribution with N-2 DF
The L0 penalty provides....
the best model given a selection criteria, but it requires fitting all submodels.
The bigger the lambda,.....
the bigger the penalty for model complexity.
The overall regression F-statistic tests the null hypothesis that
the coefficients are equal to zero
residuals
the difference between the observed response and the fitted responses
Cook's Distance
the distance between the fitted values of the model with all the data versus the fitted values of the model discarding the ith observation:
classification error rate
the probability that the new response is equal to the classifier.
How will we assess the normality assumption for the ANOVA model?
the quantile-quantile normal plot and the historgram of the residuals.
Deviance residuals
the signed square root of the log-likelihood evaluated at the saturated model when we assume that the estimate expected response is the observed response versus the fitted model.
Multicollinearity inflates...?
the standard error of the estimated coefficients
sum of square errors
the sum of square differences between the observations and the individual sample means
sum of square treatments
the sum of the square difference between the sample means of the individual samples minus the overall mean
Controlling factors
to control for bias selection in the sample. They are used as 'default' variables in order to capture more meaningful relationships.
Explanatory factors
to explain variability in the response variable; they may be included in the model even if other "similar" variables are in the model
what is the confidence interval used for?
to provide an interval estimate for the true average value of y for all members of the population with a particular value of x*
We interpret the beta in a logistic regression model in respect to?
to the odds of success
A Poisson regression model fit to a dataset with a small sample size will have a hypothesis testing procedure with more Type I errors than expected.
true
A logistic regression model may not be a good fit if the responses are correlated or if there is heterogeneity in the success that hasn't been modeled.
true
Akaike Information Criterion (AIC) is an estimate for the prediction risk.
true
An overdispersion parameter close to 1 indicates that the variability of the response is close to the variability estimated by the model.
true
The estimated regression coefficients in Poisson regression are approximate.
true The estimated parameters and their standard errors are approximate estimates.
To evaluate whether the model is a good fit or whether the assumptions hold, what should we use?
use the Pearson or Deviance residuals to evaluate whether they are normally distributed and conclude it is a good fit via hypotheses testing.
what is the prediction interval used for?
used to provide an interval estimate for a prediction of y for one member of the population with a particular value of x*
ANOVA measures...
variability between samples to the variability within a sample.
pooled variance estimator (MSE)
we compare the means, assuming the variances are the same, and estimate the variance across all samples
Overdispersion
where the variability of the response variable is larger than estimated by the model.
outliers
which are data points far from the majority of the data in both x and y or just x
canonical link function
which means that parameter estimates under logistic regression are fully efficient and tests on those parameters are better behaved for small samples
MSE measures..
within-group variability
An example of a multiple regression model is Analysis of Variance (ANOVA).
True
An indication that a higher order non linear relationship better fits the data is that the dummy variables are all, or nearly all, statistically significant
True
An overdispersion parameter close to 1 indicates that the variability of the response is close to the variability estimated by the model.
True
Analysis of Variance (ANOVA) is an example of a multiple regression model.
True
Assuming the model is a good fit, the residuals in simple linear regression have constant variance.
True
Before making statistical inference on regression coefficients, estimation of the variance of the error terms is necessary.
True
Both LASSO and ridge regression always provide greater residual sum of squares than that of simple multiple linear regression.
True
Classification is nothing else than prediction of binary responses.
True
Confounding variable is a variable that influences both the dependent variable and independent variable
True
Cook's distance (Di) measurs how much the fitted values in a MLR model change when the ith observation is removed
True
Event rates can be calculated as events per units of varying size, this unit of size is called exposure
True
For Poisson regression, we can reduce type I errors of identifying statistical significance in the regression coefficients by increasing the sample size.
True
For a MLR model to be a good fit, we need the linearity assumption to hold for at least one of the predicting variables
True
For a given predicting variable, the estimated coefficient of regression associated with it will likely be different in a model with other predicting variables or in the model with only the predicting variable alone.
True
For a linearly dependent set of predictor variables, we should not estimate a multiple linear regression model.
True
For assessing the normality assumption of the ANOVA model, we can use the quantile-quantile normal plot and the historgram of the residuals.
True
For both Logistic and Poisson Regression, the deviance residuals should approximately follow the standard normal distribution if the model is a good fit for the data
True
For large sample size data, the distribution of the test statistic, assuming the null hypothesis, is a chi-squared distribution
True
If a predicting variable is categorical with 5 categories in a linear regression model without intercept, we will include 5 dummy variables in the model.
True
If one confidence interval in the pairwise comparison includes only positive values, we conclude that the difference in means is positive, and statistically significant.
True
If one confidence interval in the pairwise comparison includes only positive values, we conclude that the difference in means is statistically significantly positive.
True
If one confidence interval in the pairwise comparison includes zero under ANOVA, we conclude that the two corresponding means are plausibly equal.
True
If response variable Y has a quadratic relationship with a predictor variable X, it is possible to model the relationship using multiple linear regression.
True
To what distribution can we derive the confidence interval from?
t-distribution
The sampling distribution of β ^ 0 is a t-distribution chi-squared distribution normal distribution None of the above
t-distribution See 1.4 Statistical Inference The distribution of β 0 is normal. Since we are using a sample and not the full population, the sampling distribution of β ^ 0 is the t-distribution.
What is the sampling distribution for individual β hat?
t-distribution with n-p-1 DF
What can we use to test for statistical significance?
t-test
If there is a group of variables among which the correlation are very high, then the Lasso...
tends to select only one variable from that group and does not care which one is selected.
What is the F-test for?
test for overall regression
T/F: If you apply linear regression under normality to count data, the assumption of constant variance still holds.
F
T/F: In Poisson regression, we also need to check for the assumption of constant variance of the error terms.
F
T/F: In multiple linear regression, a VIF value of 6 for a predictor means that 80% of the variation in that predictor can be modeled by the other predictors.
F
T/F: In multiple linear regression, if the coefficient of a quantitative predicting variable is negative, that means the response variable will decrease as this predicting variable increases.
F
T/F: In multiple linear regression, we need the linearity assumption to hold for at least one of the predicting variables
F
T/F: In simple linear regression models, we lose three degrees of freedom because of the estimation of the three model parameters β 0 , β 1 , σ 2.
F
T/F: Multicollinearity in multiple linear regression means that the rows in the design matrix are (nearly) linearly dependent.
F
T/F: Only the log-transformation of the response variable can be used when the normality assumption does not hold.
F
T/F: The binary response variable in logistic regression has a Bernoulli distribution.
F
T/F: The canonical link function for Poisson regression is the logit link function.
F
T/F: The coefficient of variation is used to evaluate goodness-of-fit.
F
T/F: The constant variance assumption is diagnosed using the histogram.
F
T/F: The error term in logistic regression has a normal distribution.
F
T/F: The log-likelihood function is a linear function with a closed-form solution.
F
T/F: The logit link function is the best link function to model binary response data because the models produced always fit the data better than other link functions.
F
T/F: The mean sum of square errors in ANOVA measures variability between groups.
F
T/F: The number of degrees of freedom of the χ 2 (chi-square) distribution for the variance estimator is N − k + 1 where k is the number of samples.
F
T/F: The number of parameters that need to be estimated in a logistic regression model with 5 predicting variables and an intercept is the same as the number of parameters that need to be estimated in a standard linear regression model with an intercept and same predicting variables.
F
T/F: The prediction of the response variable and the estimation of the mean response have the same interpretation.
F
T/F: The prediction of the response variable has the same levels of uncertainty compared with the estimation of the mean response.
F
T/F: The regression coefficients are used to measure the linear dependence between two variables.
F
T/F: The regression coefficients for the Poisson regression can be estimated in exact/closed form.
F
T/F: The sampling distribution for the estimated regression coefficients under logistic regression is approximately t-distribution.
F
T/F: The sampling distribution for the variance estimator in ANOVA is χ 2 (chi-square) regardless of the assumptions of the data.
F
T/F: The sampling distribution of the predicted response variable used in statistical inference is normal in multiple linear regression under the normality assumption.
F
T/F: The sampling distribution of the prediction of the response variable is a χ 2(chi-squared) distribution.
F
T/F: The statistical inference for linear regression under normality relies on large size of sample data.
F
T/F: We assess the assumption of constant-variance by plotting the response variable against fitted values.
F
T/F: We can perform goodness-of-fit analysis through residual diagnosis for a logistic regression without replications.
F
T/F: We interpret logistic regression coefficients with respect to the response variable.
F
T/F: When testing a subset of coefficients, deviance follows a chi-square distribution with q degrees of freedom, where q is the number of regression coefficients in the reduced model.
F
T/F: β ^ 1 is an unbiased estimator for β0.
F
T/F; A linear regression model has high predictive power if the coefficient of determination is close to 1.
F
T/F: Under logistic regression, the sampling distribution used for a coefficient estimator is a chi-square distribution.
F, derived using MLE which is a normal distribution.
A study was conducted to measure the effect of a fungicide treatment on the survival rate of botrytis blight. Botrytis blight samples were divided into 20 groups, each consisting of about 100 samples and exposed to different levels of chemicals in a fungicide. The output of a logistic regression model is below, where concS represents the concentration of a sulfur in the fungicide and concCu represents the concentration of a copper in the fungicide. Use it to answer the following multiple-choice questions. Call: glm(formula = cbind(Survived, Died) ~ concS + concCu,family = "binomial", data = data) Deviance Residuals: Min 1Q Median 3Q Max -9.5366 -2.4594 0.1223 3.9710 6.3566 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) 3.58770 0.22958 15.63 <2e-16 *** concS -4.32735 0.26518 16.32 <2e-16 *** concCu -0.27483 0.01784 15.40 <2e-16 *** Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 718.76 on 19 degrees of freedom Residual deviance: 299.43 on 17 degrees of freedom AIC: 363.53 Construct an approximate 95% confidence interval for the coefficient of concCu. (-0.322, -0.249) (-4.931, -3.724) (-4.847, -3.808) (-0.310,-0.240)
(-0.310,-0.240) [-0.27483-1.96*0.01784, -0.27483+1.96*0.01784]
Penalized or regularized regression
When we perform variable selection and estimation simultaneously.
An experiment was conducted to determine the effect of gamma radiation on the numbers of chromosomal abnormalities observed in cells. A multiple linear regression model was fitted to estimate the effect of the number of cells, amount of the radiation dose (Grays), and the rate of the radiation dose (Grays/hour) on the number of chromosomal abnormalities observed. The data frame has 27 observations. Here is the model summary and Cook's Distance plot. Coefficient Estimate SE t-value Pr(>|t|) (Intercept) -74.15392 42.24544 -1.755 0.092518 cells 0.06871 0.02196 3.129 0.004709** doseamt 41.33160 9.13907 4.523 0.000153*** doserate 20.28402 8.29071 2.447 0.022482* ---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 54.05 on X degrees of freedom Multiple R-squared: 0.5213, Adjusted R-squared: 0.4588 F-statistic: 8.348 on X and Y DF, p-value: 0.0006183 Suppose you wanted to test if the coefficient for doseamt is equal to 50. What t-value would you use for this test? 1.54 -0.948 0.692 -0.882
-0.948 t-value = (41.33160−50)/ 9.13907 = −8.6684/ 9.13907 = -0.9484991
A study was conducted to measure the effect of a fungicide treatment on the survival rate of botrytis blight. Botrytis blight samples were divided into 20 groups, each consisting of about 100 samples and exposed to different levels of chemicals in a fungicide. The output of a logistic regression model is below, where concS represents the concentration of a sulfur in the fungicide and concCu represents the concentration of a copper in the fungicide. Use it to answer the following multiple-choice questions. Call: glm(formula = cbind(Survived, Died) ~ concS + concCu,family = "binomial", data = data) Deviance Residuals: Min 1Q Median 3Q Max -9.5366 -2.4594 0.1223 3.9710 6.3566 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) 3.58770 0.22958 15.63 <2e-16 *** concS -4.32735 0.26518 16.32 <2e-16 *** concCu -0.27483 0.01784 15.40 <2e-16 *** Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 718.76 on 19 degrees of freedom Residual deviance: 299.43 on 17 degrees of freedom AIC: 363.53 Suppose you wanted to test if the coefficient for concCu is equal to -0.2. What z-value would you use for this test? 0.095 -0.073 -4.195 1.411
-4.195 (-0.27483-(-0.2))/0.01784 = -4.195
You were hired to consult on a study for the attendance behavior of high school students at two different schools. The data set you were given contains for each 316 students: the number of days he/she was absent in an academic year (daysabs), his/her math scores (math), his/her language arts scores (langarts), and whether the student is male or female (1 = male, 0 = female). A Poisson regression model was fitted to evaluate the relationship between the number of days of absence in an academic year and all the predictors. The R output for the model summary is as follows: Coefficient Estimate SE z value Pr(>|z|) (Intercept) 2.687666 0.072651 36.994 <2e-16 math -0.003523 0.001821 -1.934 0.0531 langarts -0.012152 0.001835 -6.623 3.52e-11 male -0.400921 0.048412 -8.281 <2e-16 Also, assume the average language arts scores (across all students) is 50, and the average math scores (across all students) is 45.5. For students with average math and language arts scores, how many more days on average are female students absent compared to their male counterparts? 4.8545 3.5729 2.2525 0.6697
2.2525 λ(Xmath, Xlangarts, Xmale) = e^( 2.687666−0.003523Xmath−0.012152Xlangarts−0.400921∗Xmale ) λ(Xmath = 45.5, Xlangarts = 50, Xmale = 0) = e^( 2.687666−0.003523∗(45.5)−0.012152∗(50)−0.400921∗(0) = 6.819386 ) λˆ(Xmath = 45.5, Xlangarts = 50, Xmale = 1) = e ^(2.687666−0.003523∗(45.5)−0.012152∗(50)−0.400921∗(1) = 4.566963 ) λ(Xmath = 45.5, Xlangarts = 50, Xmale = 0) − λ(Xmath = 45.5, Xlangarts = 50, Xmale = 1) = 2.252423
Box-cox transformation
A common transformation is the power transformation y to the lambda used to improve the normality and/or constant variance assumption.
Influential points
A data point that is far from the mean both x and y and change the value of the estimated parameters significantly. It can change the statistical significance, it can change the magnitude. It can change even the sign.
An experiment was conducted to determine the effect of gamma radiation on the numbers of chromosomal abnormalities observed in cells. A multiple linear regression model was fitted to estimate the effect of the number of cells, amount of the radiation dose (Grays), and the rate of the radiation dose (Grays/hour) on the number of chromosomal abnormalities observed. The data frame has 27 observations. Here is the model summary and Cook's Distance plot. Coefficient Estimate SE t-value Pr(>|t|) (Intercept) -74.15392 42.24544 -1.755 0.092518 cells 0.06871 0.02196 3.129 0.004709** doseamt 41.33160 9.13907 4.523 0.000153*** doserate 20.28402 8.29071 2.447 0.022482* ---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 54.05 on X degrees of freedom Multiple R-squared: 0.5213, Adjusted R-squared: 0.4588 F-statistic: 8.348 on X and Y DF, p-value: 0.0006183 For an F-test of overall significance of the regression model, what degrees of freedom would be used? 3 , 24 2, 27 3, 23 2, 23
3, 23 The numerator degrees of freedom (ndf) is equal to p and the denominator degrees of freedom (ddf) is equal to n-p-1, where n: number of observations and p: number of predictors. Hence, ndf = 3 and ddf = 27-3-1 = 23
The following output was captured from the summary output of a simple linear regression model that relates the duration of an eruption with the waiting time since the previous eruption. Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -1.374016 A -1.70 0.045141 * waiting 0.043714 0.011098 B 0.000052 *** --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 0.4965 on 270 degrees of freedom Multiple R-squared: 0.8115, Adjusted R-squared: 0.8108 F-statistic: 1162 on 1 and 270 DF, p-value: < 2.2e-16 Using the table above, what is the t-value of the coefficient for waiting, labeled B, and rounded to three decimal places? 3.939 3.931 3.935 None of the above
3.939 See 1.4 Statistical Inference t-value = Estimate /Std.Err= 0.043714/0.011098 = 3.939
You were hired to consult on a study for the attendance behavior of high school students at two different schools. The data set you were given contains for each 316 students: the number of days he/she was absent in an academic year (daysabs), his/her math scores (math), his/her language arts scores (langarts), and whether the student is male or female (1 = male, 0 = female). A Poisson regression model was fitted to evaluate the relationship between the number of days of absence in an academic year and all the predictors. The R output for the model summary is as follows: Coefficient Estimate SE z value Pr(>|z|) (Intercept) 2.687666 0.072651 36.994 <2e-16 math -0.003523 0.001821 -1.934 0.0531 langarts -0.012152 0.001835 -6.623 3.52e-11 male -0.400921 0.048412 -8.281 <2e-16 Also, assume the average language arts scores (across all students) is 50, and the average math scores (across all students) is 45.5. What is the expected number of days missed for a female student with a langarts of 48 and a math score of 50 based on the model? 6.8773 1.9106 6.6363 4.5251
6.8773 λ(Xmath, Xlangarts, Xmale) = e^( 2.687666−0.003523Xmath−0.012152Xlangarts−0.400921Xmale) λ(Xmath = 50, Xlangarts = 48, Xmale = 0) = e^( 2.687666−0.003523∗50−0.012152∗48−0.400921∗0) = 6.877258
influential points
A data point that is far from the mean of both the x's and the y's, because they are influencing the fit of the regression.
An experiment was conducted to determine the effect of gamma radiation on the numbers of chromosomal abnormalities observed in cells. A multiple linear regression model was fitted to estimate the effect of the number of cells, amount of the radiation dose (Grays), and the rate of the radiation dose (Grays/hour) on the number of chromosomal abnormalities observed. The data frame has 27 observations. Here is the model summary and Cook's Distance plot. Coefficient Estimate SE t-value Pr(>|t|) (Intercept) -74.15392 42.24544 -1.755 0.092518 cells 0.06871 0.02196 3.129 0.004709** doseamt 41.33160 9.13907 4.523 0.000153*** doserate 20.28402 8.29071 2.447 0.022482* ---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 54.05 on X degrees of freedom Multiple R-squared: 0.5213, Adjusted R-squared: 0.4588 F-statistic: 8.348 on X and Y DF, p-value: 0.0006183 Calculate the Sum of Squared Regression from the model summary. 17,484.25 73,163.60 67,181.18 55,284.40
73,163.60 You could calculate this value in several ways. This is one possible way. Fstat = MSReg/MSE = (SSReg/p)/MSE SSReg = Fstat ∗ MSE ∗ p = (8.348)(54.052 )(3) = 73,163.60
In Poisson regression, A) We model the log of the expected response variable not the expected log response variable. B) We use the ordinary least squares to fit the model. C) There is an error term. D) None of the above.
A
In logistic regression, A) The estimation of the regression coefficients is based on maximum likelihood estimation. B) We can derive exact (close form expression) estimates for the regression coefficients. C) The estimations of the regression coefficients is based on minimizing the sum of least squares. D) All of the above.
A
The mean squared errors (MSE) measures: A) The within-treatment variability. B) The between-treatment variability. C) The sum of the within-treatment and between-treatment variability. D) None of the above.
A
The objective of the residual analysis is: A) To evaluate departures from the model assumptions B) To evaluate whether the means are equal. C) To evaluate whether only the normality assumptions holds. D) None of the above.
A
The pooled variance estimator is: A) The sample variance estimator assuming equal variances. B) The variance estimator assuming equal means and equal variances. C) The sample variance estimator assuming equal means. D) None of the above.
A
Which is correct? A) The regression coefficients can be estimated only if the predicting variables are not linearly dependent. B) The estimated regression coefficient beta hat i is interpreted as the change in the response variable associated with one unit of change in the i-th predicting variable. C) The estimated regression coefficients will be the same under marginal and conditional model; only their interpretation is not. D) Causality is the same as association in interpreting the relationship between the response and predicting variables.
A
Which one is correct? A) We use a chi-square testing procedure to test whether a subset of regression coefficients are zero in Poisson regression. B) The test for subsets of regression coefficients is a goodness of fit test. C) The test for subsets of regression coefficients is reliable for small sample data in Poisson regression. D) None of the above.
A
A study was conducted to measure the effect of a fungicide treatment on the survival rate of botrytis blight. Botrytis blight samples were divided into 20 groups, each consisting of about 100 samples and exposed to different levels of chemicals in a fungicide. The output of a logistic regression model is below, where concS represents the concentration of a sulfur in the fungicide and concCu represents the concentration of a copper in the fungicide. Use it to answer the following multiple-choice questions. Call: glm(formula = cbind(Survived, Died) ~ concS + concCu,family = "binomial", data = data) Deviance Residuals: Min 1Q Median 3Q Max -9.5366 -2.4594 0.1223 3.9710 6.3566 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) 3.58770 0.22958 15.63 <2e-16 *** concS -4.32735 0.26518 16.32 <2e-16 *** concCu -0.27483 0.01784 15.40 <2e-16 *** Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 718.76 on 19 degrees of freedom Residual deviance: 299.43 on 17 degrees of freedom AIC: 363.53 Interpret the coefficient for concCu. A 1-unit increase in the concentration of copper decreases the odds of botrytis blight surviving by 0.27483 holding sulfer constant. A 1-unit increase in the concentration of copper decreases the number of samples of botrytis blight surviving by 0.27483 holding sulfer constant. A 1-unit increase in the concentration of copper decreases the log odds of botrytis blight surviving by 0.27483 holding sulfer constant. A 1-unit increase in the concentration of copper decreases the probability of botrytis blight surviving by 0.27483 holding sulfer constant.
A 1-unit increase in the concentration of copper decreases the log odds of botrytis blight surviving by 0.27483 holding sulfer constant.
Assuming that the data are normally distributed, the estimated variance has the following sampling distribution under the simple linear model: A. Chi-square with n-2 degrees of freedom B. T-distribution with n-2 degrees of freedom C. Chi-square with n degrees of freedom D. T-distribution with n degrees of freedom
A. Chi-square with n-2 degrees of freedom 1.1 - Knowledge Check 1
The estimators of the linear regression model are derived by: A. Minimizing the sum of squared differences between observed and expected values of the response variable. B. Maximizing the sum of squared differences between observed and expected values of the response variable. C. Minimizing the sum of absolute differences between observed and expected values of the response variable. D. Maximizing the sum of absolute differences between observed and expected values of the response variable.
A. Minimizing the sum of squared differences between observed and expected values of the response variable. 1.1 - Knowledge Check 1
Which one is correct? A. The regression coefficients can be estimated only if the predicting variables are not linearly dependent. B. The estimated regression coefficient 𝛽∧𝑖 is interpreted as the change in the response variable associated with one unit of change in the i-th predicting variable . C. The estimated regression coefficients will be the same under marginal and conditional model, only their interpretation is not. D. Causality is the same as association in interpreting the relationship between the response and the predicting variables.
A. The regression coefficients can be estimated only if the predicting variables are not linearly dependent. 3.1 - Knowledge Check 1
The pooled variance estimator is: A. The variance estimator assuming equal variances. B. The variance estimator assuming equal means and equal variances. C. The sample variance estimator assuming equal means. D. None of the above.
A. The variance estimator assuming equal variances. 2.1 - Knowledge Check 1
The mean squared errors (MSE) measures: A. The within-treatment variability. B. The between-treatment variability. C. The sum of the within-treatment and between-treatment variability. D. None of the above.
A. The within-treatment variability. 2.1 - Knowledge Check 2
The objective of the residual analysis is A. To evaluate goodness of fit B. To evaluate whether the means are equal. C. To evaluate whether only the normality assumptions holds. D. None of the above.
A. To evaluate goodness of fit 2.2 - Knowledge Check 3
We detect departure from the assumption of constant variance A. When the residuals increase as the fitted values increase also. B. When the residuals vs fitted are scattered randomly around the zero line. C. When the histogram does not have a symmetric shape. D. All of the above.
A. When the residuals increase as the fitted values increase also. 1.3 - Knowledge Check 4
The sampling distribution of β ^ 0 is a: A. t-distribution B. chi-squared distribution C. normal distribution D. None of the above
A. t-distribution See 1.4 Statistical Inference The distribution of β 0 is normal. Since we are using a sample and not the full population, the sampling distribution of β ^ 0 is the t-distribution.
Multiple Linear Regression
Assumptions: - The deviances or error terms have 0 mean - Constant variance - They are independent. Statistical Inference: - We also need to assume that the error terms are normally distributed
Multiple Linear Regression
Assumptions: We assume that the error terms epsilon_i have 0 mean and constant variance, and they're independent. We also assumed that error terms are normally distributed The parameter is defining the regression line, beta 0, beta 1, to beta p are parameters. But we also have the additional parameter, the variance of the error terms, denoted with sigma squared These parameters are unknown but estimated based on the observed data using the ordinary least squares approach Statistical inference on the regression coefficients using the sampling distribution of the estimated regression coefficients that is the T distribution with n-p-1 degrees of freedom
Comparing cross-validation methods, A) The random sampling approach is more computational efficient that leave-one-out cross validation. B) In K-fold cross-validation, the larger K is, the higher the variability in the estimation of the classification error is. C) Leave-one-out cross validation is a particular case of the random sampling cross-validation. D) None of the above.
B
In logistic regression, A) The hypothesis test for subsets of coefficients is a goodness of fit test. B) The hypothesis test for subsets of coefficients is approximate; it relies on large sample size. C) We can use the partial F test for testing whether a subset of coefficients are all zero. D) None of the above.
B
The estimated versus predicted regression line for a given x*: A) Have the same variance B) Have the same expectation C) Have the same variance and expectation D) None of the above
B
The objective of the pairwise comparison is: A) To find which means are equal. B) To identify the statistically significantly different means. C) To find the estimated means which are greater or lower than other. D) None of the above.
B
The total sum of squares divided by N-1 is: A) The mean sum of squared errors B) The sample variance estimator assuming equal means and equal variances C) The sample variance estimator assuming equal variances. D) None of the above.
B
Which one is correct? A) The logit link function is the only link function that can be used for modeling binary response data. B) Logistic regression models the probability of a success given a set of predicting variables. C) The interpretation of the regression coefficients in logistic regression is the same as for standard linear regression assuming normality. D) None of the above.
B
Which one is correct? A) We can evaluate the goodness of fit a model using the testing procedure of the overall regression. B) In applying the deviance test for goodness of fit in logistic regression, we seek large p-values, that is, not reject the null hypothesis. C) There is no error term in logistic regression and thus we cannot perform a goodness of fit assessment. D) None of the above.
B
The fitted values are defined as: A. The difference between observed and expected responses. B. The regression line with parameters replaced with the estimated regression coefficients. C. The regression line. D. The response values.
B. The regression line with parameters replaced with the estimated regression coefficients. 1.1 - Knowledge Check 1
The total sum of squares divided by N-1 is A. The mean sum of squared errors B. The sample variance estimator assuming equal means and equal variances C. The sample variance estimator assuming equal variances. D. None of the above.
B. The sample variance estimator assuming equal means and equal variances 2.1 - Knowledge Check 2
The objective of the pairwise comparison is A. To find which means are equal. B. To identify the statistically significantly different means. C. To find the estimated means which are greater or lower than other. D. None of the above.
B. To identify the statistically significantly different means. 2.2 - Knowledge Check 3
To test if a coefficient is less than a critical value, C, we conduct a one-sided test on the _________ tail of a ___________ distribution. A. left, normal B. left, t C. right, normal D. right, t E. None of the above
B. left, t See 1.4 Statistical Inference "For β 1 greater than zero we're interested on the right tail of the distribution of the β ^ 1."
The F-test is a _________ tailed test with ______ and ______ degrees of freedom. A. one, k, N-1 B. one, k-1, N-k C. two, k-1, N-k D. two, k, N-1 E. None of the above.
B. one, k-1, N-k See 2.4 Test for Equal Means The F-test is a one tailed test that has two degrees of freedom, namely k − 1 and N − k.
what are the model parameters to be estimated in MLR?
B0 (intercept), B1-Bp, and sigma squared
what are the 3 parameters we estimated in regression?
B0, B1, sigma squared (variance of the one pop.)
In SLR, we are interested in the behavior of which parameter?
B1
what does 'statistical significance' mean?
B1 is statistically different from zero.
AIC vs BIC
BIC is similar to AIC except that the complexity is penalized by log(n)/2
Which is more computationally expensive, forward or backward?
Backward
2. Which is correct? A) A multiple linear regression model with p predicting variables but no intercept has p model parameters. B) The interpretation of the regression coefficients is the same whether or not interaction terms are included in the model. C) Multiple linear regression is a general model encompassing both ANOVA and simple linear regression. D) None of the above.
C
In logistic regression: A) We can perform residual analysis for response data with or without replications. B) Residuals are derived as the fitted values minus the observed responses. C) The sampling distribution of the residual is approximately normal distribution if the model is a good fit. D) All of the above.
C
The assumption of normality: A) It is needed for deriving the estimators of the regression coefficients. B) It is not needed for linear regression modeling and inference. C) It is needed for the sampling distribution of the estimators of the regression coefficients and hence for inference. D) It is needed for deriving the expectation and variance of the estimators of the regression coefficients.
C
The estimators for the regression coefficients are: A) Biased but with small variance B) Unbiased under normality assumptions but biased otherwise. C) Unbiased regardless of the distribution of the data.
C
The variability in the prediction comes from: A) The variability due to a new measurement. B) The variability due to estimation. C) The variability due to a new measurement and due to estimation. D) None of the above.
C
Using the R statistical software to fit a logistic regression, A) We can use the lm() command. B) The input of the response variable is exactly the same if the binary response data are with or without replications. C) We can obtain both the estimates and the standard deviations of the estimates for the regression coefficients. D) None of the above.
C
Which is correct? A) If we reject the test of equal means, we conclude that all treatment means are not equal. B) If we do not reject the test of equal means, we conclude that means are definitely all equal C) If we reject the test of equal means, we conclude that some treatment means are not equal. D) None of the above.
C
Link Functions
C-log Function: This has very long tails, meaning that it works best in extremely skewed distributions Probit Function: This is the inverse of the CDF of a standard normal distribution. This fits data with least-heavy tails among the three S shaped functions. This would work well when the probabilities are all concentrated within a small range Logit Function: This is what is called the canonical link function, which means that parameter estimates under logistic regression are fully efficient and tests on those parameters are better behaved for small samples. The interpretations of regression coefficients in terms of log odds is possible with a logit function but not other S-shape functions
Which is correct? A. If we reject the test of equal means, we conclude that all treatment means are not equal. B. If we do not reject the test of equal means, we conclude that means are definitely all equal C. If we reject the test of equal means, we conclude that some treatment means are not equal. D. None of the above.
C. If we reject the test of equal means, we conclude that some treatment means are not equal. 2.1 - Knowledge Check 2
The assumption of normality: A. It is needed for deriving the estimators of the regression coefficients. B. It is not needed for linear regression modeling and inference. C. It is needed for the sampling distribution of the estimators of the regression coefficients and hence for inference. D. It is needed for deriving the expectation and variance of the estimators of the regression coefficients.
C. It is needed for the sampling distribution of the estimators of the regression coefficients and hence for inference. 1.2 - Knowledge Check 2
Which one is correct? A. A multiple linear regression model with p predicting variables but no intercept has p model parameters. B. The interpretation of the regression coefficients is the same whether or not interaction terms are included in the model. C. Multiple linear regression is a general model encompassing both ANOVA and simple linear regression. D. None of the above.
C. Multiple linear regression is a general model encompassing both ANOVA and simple linear regression. 3.1 - Knowledge Check 1
The variability in the prediction comes from: A. The variability due to a new measurement. B. The variability due to estimation C. The variability due to a new measurement and due to estimation. D. None of the above.
C. The variability due to a new measurement and due to estimation. 1.2 - Knowledge Check 3
The alternative hypothesis of ANOVA can be stated as, A. the means of all pairs of groups are different B. the means of all groups are equal C. the means of at least one pair of groups is different D. None of the above
C. the means of at least one pair of groups is different See 2.4 Test for Equal Means "Using the hypothesis testing procedure for equal means, we test: The null hypothesis, which that the means are all equal (mu 1 = mu 2...=mu k) versus the alternative hypothesis, that some means are different. Not all means have to be different for the alternative hypothesis to be true -- at least one pair of the means needs to be different."
marginal relationship
Capturing the association of a predicting variable to the response variable marginally, i.e. without consideration of other factors.
Marginal Relationship
Capturing the association of a predicting variable to the response variable without consideration of other factors
Under the null hypothesis of good fit, the test statistic's (sum of squared deviances) distribution and DOF is...?
Chi square with n-p-1 DF
Assuming that the data are normally distributed, under the simple linear model, the estimated variance has the following sampling distribution:
Chi-square with n-2 degrees of freedom
You were hired to consult on a study for the attendance behavior of high school students at two different schools. The data set you were given contains for each 316 students: the number of days he/she was absent in an academic year (daysabs), his/her math scores (math), his/her language arts scores (langarts), and whether the student is male or female (1 = male, 0 = female). A Poisson regression model was fitted to evaluate the relationship between the number of days of absence in an academic year and all the predictors. The R output for the model summary is as follows: Coefficient Estimate SE z value Pr(>|z|) (Intercept) 2.687666 0.072651 36.994 <2e-16 math -0.003523 0.001821 -1.934 0.0531 langarts -0.012152 0.001835 -6.623 3.52e-11 male -0.400921 0.048412 -8.281 <2e-16 Also, assume the average language arts scores (across all students) is 50, and the average math scores (across all students) is 45.5. The approximated distribution of the residual deviance is ____ with ____ degrees of freedom. Normal, 315 Chi-squared, 312 Chi-squared, 315 t, 312
Chi-squared, 312 The approximated distribution of the residual deviance is Chi-square with n-p-1 degrees of freedom. In this example n = 316 and p = 3 ; Hence df= 312.
In evaluating a multiple linear model: A) The F test is used to evaluate the overall regression. B) The coefficient of variation is interpreted as the percentage of variability in the response variable explained by the model. C) Residual analysis is used for goodness of fit assessment. D) All of the above.
D
In the presence of near multicollinearity: A) The coefficient of variation decreases. B) The regression coefficients will tend to be identified as statistically significant even if they are not. C) The prediction will not be impacted. D) None of the above.
D
Logistic regression is different from standard linear regression in that: A) It does not have an error term B) The response variable is not normally distributed. C) It models probability of a response and not the expectation of the response. D) All of the above.
D
Logistic regression is different from standard linear regression in that: A) The sampling distribution of the regression coefficient is approximate. B) A large sample data is requirded for making accurate statistical inferences. C) A normal sampling distribution is used instead of a t-distribution for statistical inference. D) All of the above.
D
Poisson regression can be used: A) To model count data. B) To model rate response data. C) To model response data with a Poisson distribution. D) All of the above.
D
Residual analysis in Poisson regression can be used: A) To evaluate goodness of fit of the model. B) To evaluate whether the relationship between the log of the expected response and the predicting variables is linear. C) To evaluate whether the data are uncorrelated. D) All of the above.
D
The estimators for the regression coefficients are: A) Biased but with small variance B) Unbiased under normality assumptions but biased otherwise C) Biased regardless of the distribution of the data. D) Unbiased regardless of the distribution of the data.
D
The objective of multiple linear regression is: A) To predict future new responses B) To model the association of explanatory variables to a response variable accounting for controlling factors. C) To test hypotheses using statistical inference on the model. D) All of the above.
D
The sampling distribution of the estimated regression coefficients is: A) Centered at the true regression parameters. B) The t-distribution assuming that the variance of the error term is unknown and replaced by its estimate. C) Dependent on the design matrix. D) All of the above.
D
We can test for a subset of regression coefficients: A) Using the F-statistic test of the overall regression. B) Only if we are interested in whether additional explanatory variables should be considered in addition to the controlling variables. C) To evaluate whether all regression coefficients corresponding to the predicting variables excluded from the reduced model are statistically significant. D) None of the above.
D
When do we use transformations? A) If the linearity assumption with respect to one or more predictors does not hold, then we use transformations of the corresponding predictors to improve on this assumption. B) If the normality assumption does not hold, we transform the response variable, commonly using the Box-Cox transformation. C) If the constant variance assumption does not hold, we transform the response variable. D) All of the above.
D
When we do not have a good fit in generalized linear models, it may be that: A) We need to transform some of the predicting variables or to include other variables. B) The variability of the expected rate is higher than estimated. C) There may be leverage point that need to explored further. D) All of the above.
D
Which are all the model parameters in ANOVA? A) The means of the k populations. B) The sample means of the k populations. C) The sample means of the k samples. D) None of the above.
D
Which is correct? A) Prediction translates into classification of a future binary response in logistic regression. B) In order to perform classification in logistic regression, we need to first define a classifier for the classification error rate. C) One common approach to estimate the classification error is cross-validation. D) All of the above.
D
Which one correctly characterizes the sampling distribution of the estimated variance? A) The estimated variance of the error term has a chi-squared distribution regardless of the distribution assumption of the error terms. B) The number of degrees of freedom for the chi-squared distribution of the estimated variance is n - p - 1 for a model without an intercept. C) The sampling distribution of the mean squared error is different of that of the estimated variance. D) None of the above.
D
Which one is correct? A) The prediction intervals need to be corrected for simultaneous inference when multiple predictions are made jointly. B) The prediction intervals are centered at the predicted value. C) The sampling distribution of the prediction of a new response is a t-distribution. D) All of the above.
D
Which one is correct? A) The estimated regression coefficients and their standard deviations are approximate not exact in Poisson regression. B) We use the glm() R command to fit a Poisson linear regression. C) The interpretation of the estimated regression coefficients is in terms of the ratio of the response rates. D) All of the above.
D
Which one is correct? A) The residuals have constant variance for the multiple linear regression model. B) The residuals vs. fitted can be used to assess the assumption of independence. C) The residuals have a t-distribution if the error term is assumed to have a normal distribution. D) None of the above.
D
Which one is correct? A) The standard normal regression, the logistic regression and the Poisson regression are all falling under the generalized linear model framework. B) If we were to apply a standard normal regression to response data with a Poisson distribution, the constant variance assumption would not hold. C) The link function for the Poisson regression is the log function. D) All of the above.
D
What is the difference between LASSO and Elastic Net?
Elastic Net has an additional penalty just like the one used in ridge regression.
In evaluating a simple linear model A. There is a direct relationship between the coefficient of determination and the correlation between the predicting and response variables. B. The coefficient of determination is interpreted as the percentage of variability in the response variable explained by the model. C. Residual analysis is used for goodness of fit assessment. D. All of the above.
D. All of the Above 1.3 - Knowledge Check 4
In evaluating a multiple linear model, A. The F test is used to evaluate the overall regression. B. The coefficient of determination is interpreted as the percentage of variability in the response variable explained by the model. C. Residual analysis is used for goodness of fit assessment. D. All of the above.
D. All of the Above 3.3 - Knowledge Check 4
When do we use transformations? A. If the linearity assumption with respect to one or more predictors does not hold, then we use transformations of the corresponding predictors to improve on this assumption. B. If the normality assumption does not hold, we transform the response variable, commonly using the Box-Cox transformation. C. If the constant variance assumption does not hold, we transform the response variable. D. All of the above.
D. All of the Above 3.3 - Knowledge Check 4
The objective of multiple linear regression is A. To predict future new responses B. To model the association of explanatory variables to a response variable accounting for controlling factors. C. To test hypothesis using statistical inference on the model. D. All of the above.
D. All of the above. 3.1 - Knowledge Check 1
The sampling distribution of the estimated regression coefficients is A. Centered at the true regression parameters. B. The t-distribution assuming that the variance of the error term is unknown an replaced by its estimate. C. Dependent on the design matrix. D. All of the above.
D. All of the above. 3.2 - Knowledge Check 2
In the presence of near multicollinearity, A. The coefficient of determination decreases. B. The regression coefficients will tend to be identified as statistically significant even if they are not. C. The prediction will not be impacted. D. None of the above.
D. None of the Above 3.3 - Knowledge Check 4
Which one is correct? A. The residuals have constant variance for the multiple linear regression model. B. The residuals vs fitted can be used to assess the assumption of independence. C. The residuals have a t-distribution distribution if the error term is assumed to have a normal distribution. D. None of the above.
D. None of the Above 3.3 - Knowledge Check 4
Which one is correct? A. If a departure from normality is detected, we transform the predicting variable to improve upon the normality assumption. B. If a departure from the independence assumption is detected, we transform the response variable to improve upon this assumption. C. The Box-Cox transformation is commonly used to improve upon the linearity assumption. D. None of the above
D. None of the above. 1.3 - Knowledge Check 4
Which are all the model parameters in ANOVA? A. The means of the k populations. B. The sample means of the k populations. C. The sample means of the k samples. D. None of the above.
D. None of the above. 2.1 - Knowledge Check 1
Which one correctly characterizes the sampling distribution of the estimated variance? A. The estimated variance of the error term has a 𝜒2distribution regardless of the distribution assumption of the error terms. B. The number of degrees of freedom for the 𝜒2 distribution of the estimated variance is n-p-1 for a model without intercept. C. The sampling distribution of the mean squared error is different of that of the estimated variance. D. None of the above.
D. None of the above. 3.1 - Knowledge Check 1
We can test for a subset of regression coefficients A. Using the F statistic test of the overall regression. B. Only if we are interested whether additional explanatory variables should be considered in addition to the controlling variables. C. To evaluate whether all regression coefficients corresponding to the predicting variables excluded from the reduced model are statistically significant. D. None of the above.
D. None of the above. 3.2 - Knowledge Check 2
You were hired to consult on a study for the attendance behavior of high school students at two different schools. The data set you were given contains for each 316 students: the number of days he/she was absent in an academic year (daysabs), his/her math scores (math), his/her language arts scores (langarts), and whether the student is male or female (1 = male, 0 = female). A Poisson regression model was fitted to evaluate the relationship between the number of days of absence in an academic year and all the predictors. The R output for the model summary is as follows: Coefficient Estimate SE z value Pr(>|z|) (Intercept) 2.687666 0.072651 36.994 <2e-16 math -0.003523 0.001821 -1.934 0.0531 langarts -0.012152 0.001835 -6.623 3.52e-11 male -0.400921 0.048412 -8.281 <2e-16 Also, assume the average language arts scores (across all students) is 50, and the average math scores (across all students) is 45.5. How does an increase in 1 unit in langarts affect the expected number of days missed, given that the other predictors in the model are held constant? Increase by 0.012152 days Increase by 0.9879 days Increase by 1.22% Decrease by 1.21%
Decrease by 1.21% The estimated coefficient for langarts is -0.012152. A one unit increase in langarts gives us e −0.012152 = 0.9879215. In terms of percentages, this should be interpreted as the expected number of days missed decreasing by 1.21% (1-0.9879215). Hence, given that the other predictors in the model are held constant, a one unit increase in langarts results in the expected number of days missed decreasing by 1.21%, holding all other predictors constant.
Deviance Residuals - Poisson Regression
Deviance residuals are the sign square root of the likelihood evaluated the saturated model when we assume that the estimated expected response is the observed response versus the fitted model Normal Distribution
Causality is the same as association in interpreting the relationship between the response and the predicting variables.
False
For a multiple regression model, both the true errors LaTeX: \epsilon ϵ and the estimated residuals LaTeX: \hat \epsilon ϵ ^ have a constant mean and a constant variance.
False
For estimating confidence intervals for the regression coefficients, the sampling distribution used is a normal distribution.
False
For the model LaTeX: y=\beta_0+\beta_1x_1+...+\beta_px_p+\epsilon y = β 0 + β 1 x 1 + ... + β p x p + ϵ , where LaTeX: \epsilon\sim N(0,\sigma^2) ϵ ∼ N ( 0 , σ 2 ) , there are p+1 parameters to be estimated
False
Given a categorial predictor with 4 categories in a linear regression model with intercept, 4 dummy variables need to be included in the model.
False
If a departure from normality is detected, we transform the predicting variable to improve upon the normality assumption.
False
If a departure from the independence assumption is detected, we transform the response variable to improve upon this assumption.
False
The sampling distribution for the variance estimator in simple linear regression is χ 2 (chi-squared) regardless of the assumptions of the data. (T/F)
False See 1.2 Estimation Method "The sampling distribution of the estimator of the variance is chi-squared, with n - 2 degrees of freedom (more on this in a moment). This is under the assumption of normality of the error terms."
We assess the constant variance assumption by plotting the error terms, ϵ i, against fitted values. (T/F)
False See 1.2 Estimation Method "We use ϵ ^ i as proxies for the deviances or the error terms. We don't have the deviances because we don't have β 0 and β 1.
The simple linear regression coefficient, β ^ 0, is used to measure the linear relationship between the predicting and response variables. (T/F)
False See 1.2 Estimation Method β ^ 0 is the intercept and does not tell us about the relationship between the predicting and response variables.
β ^ 1 is an unbiased estimator for β 0.
False See 1.4 Statistical Inference "What that means is that β ^ 1 is an unbiased estimator for β 1." It is not an unbiased estimator for β 0.
β ^ 1 is an unbiased estimator for β 0. (T/F)
False See 1.4 Statistical Inference "What that means is that β ^ 1 is an unbiased estimator for β 1." It is not an unbiased estimator for β 0.
The p-value is a measure of the probability of rejecting the null hypothesis. (T/F)
False See 1.5 Statistical Inference Data Example "p-value is a measure of how rejectable the null hypothesis is... It's not the probability of rejecting the null hypothesis, nor is it the probability that the null hypothesis is true."
For a multiple linear regression model to be a good fit, we need the linearity assumption to hold for only one of the predicting variables. (T/F)
False See Lesson 3.11: Assumptions and diagnostics In multiple linear regression, we need the linearity assumption to hold for all of the predicting variables, for the model to be a good fit. "For example, if the linearity does not hold with one or more predicting variables, then we could transform the predicting variables to improve the linearity assumption."
Given a quantitative predicting variable and a qualitative predicting variable with 7 categories in a linear regression model with intercept, 7 dummy variables need to be included in the model.
False See Lesson 3.2: Basic Concepts We only need 7 dummy variables. "When we have qualitative variables with k levels, we only include k-1 dummy variables if the regression model has an intercept."
In MLR, the adjusted R2 can be used to compare models, and its value will be always greater than or equal to that of R2
False - Adjusted R2 will be always less than or equal to R2
The threshold to calculate the classification error rate of a logistic regression model should always be set at 0.5
False - Althought 0.5 is common value, the threshold is problem-dependent and its value should be tuned.
In a MLR model, an observation should always be discarded when its Cook's distance is greater than 4/n (n=sample size)
False - An observation should not be discarded just because it is found to be an outlier. We must investigate the nature of the outlier before deciding to discard it.
If the constant variance assumption does not hold in MLR, we apply Box-Cox transformation to the predicting variables
False - Apply Box-Cox to the response (y) not predicting variables.
Hypothesis testing for Poisson regression can be done on small sample sizes
False - Approximation of normal distribution needs large sample sizes, so does hypothesis testing.
Bayesian information criterion (BIC) penalizes for complexity of the model more than both leave one out CV and Mallow's Cp statistic
False - BIC penalizes complexity more than other approaches
LASSO regression will always select the same number or more predicting variables than Ridge and Elastic Net regression
False - Because LASSO can eliminate predicting variables using the penalty while the Ridge and Elastic Net retain coefficients, LASSO will have the same number or LESS predicting variables
The number of parameters that need to be estimated in a Logistic Regression model with 6 predicting variables and an intercept is the same as the number of parameters that need to be estimated in a standard Linear Regression model with an intercept and same predicting variable
False - Because it doesn't have the error term
Multicollinearity in MLR means that the rows in the design matrix are linearly dependent.
False - Columns
Multicollinearity in MLR means that the columns in the design matrix are linearly independent.
False - Columns are actually linearly DEPENDENT
Elastic net often underperforms LASSO regression in terms of prediction accuracy because it considers both L1 and L2 penalties together
False - Elastic Net often outperforms LASSO in terms of accuracy. The difference between LASSO and Elastic Net is the addition of penalty just like the one used in Ridge Regression. By considering both, L1 and L2, we have the advantages of both LASSO and Ridge Regression.
Since there are no error terms in a Poisson model, we cannot perform residual analysis for evaluating the model's GOF.
False - For Poisson Regression, we can perform residual analysis although there is not an error term.
In MLR, the coefficient of determination is used to evaluate GOF
False - GOF in MLR through model structure and assumptions
Multiplying a variable by 10 in LASSO regression, decreases the chance that the coefficient of this variable is nonzero.
False - I am not sure why anyone would think this would be true.
In Poisson Regression, the expectation of the response variable given the predictors is equal to the linear combination of the predicting variables.
False - In Poisson Regression, the expectation of the response variable given the predictors is equal to the exponential of the linear combination of the predicting variables.
In Poisson regression, we assume a nonlinear relationship between the log rate and the predicting variables.
False - In Poisson Regression, we assume a linear relationship between the log rate and the predicting variables.
The mean square prediction error(MSPE) is a robust prediction accuracy measurement for a OLS model regardless of the characteristics of the dataset.
False - MPSE is appropriate for evaluating prediction accuracy for a linear model estimated using least squares, but it depends on the scale of the response data, and thus is sensitive to outliers.
If the outcome variable is quantitative and all explanatory variables take values 0 or 1, a logistic regression model is most appropriate.
False - More research is necessary to determine the correct model.
From the binomial approximation with a normal distribution using the central limit theorem, the Pearson residuals have an approximately standard chi-squared distribution.
False - Normal distribution
The assumption of constant variance will always hold for standard Linear Regression models with Poisson distributed response data.
False - One common problem with fitting a linear regression model to Poisson data is the departure from the assumption of Constant variance.
If a Poisson regression model does not have a good fit, the relationship between the log of the expected rate and the predicting variables might not be linear
False - Only one way (Non linear relationship causes Poisson not a good fit). Not the other way around
The presence of certain types of outliers can impact the statistical significance of some of the regression coefficients of a MLR model
False - Outliers that are influential can impact statistical significance of a beta parameters.
Parameter tuning is not recommended as part of the sub-sampling approach for addressing the p-value problem with large samples.
False - Parameter tuning is recommended as part of the sub-sampling approac
The F-test can be used to test for the overall regression in Poisson regression.
False - Poisson uses Chi-squared test to test for overall regression.
Another criteria for variable selection is cross validation which is a direct measure of explanatory power.
False - Predictive power
Forward stepwise variable selection starts with the simpler model and select the predicting variable that increases the R2 the most, unless the R2 cannot be increased any further by adding variables
False - R2 is not compared during stepwise variable selection. Variables are selected if they reduced the AIC or BIC of a model.
In Logistic Regression, R2 could be used as a measure of explained variation in the response variable.
False - R2 is used as explained variability (not variation). In logistics regression, the response variable is binary therefore R2 doesn't measure explained variation.
R2 decreases as more predictors are added to a MLR model, given that the predictors added are unrelated to the response variable
False - R2 never decreases as more predictors are added to MLR.
Random sampling is computationally less expensive than the K-fold cross validation
False - Random sampling is computationally more expensive than the K-fold CV, with no clear advantage in terms of accuracy of the estimation classification error. K fold CV is preferred at least from a computation standpoint.
It is not required to standardize or rescale the predicting variables when performing regularized regression
False - Regularized regression requires standardization or scaling of the predicting variables.
The p-value of the test computed as the left tail of the chi-squared distribution
False - Right tail
When assessing GOF for a Logistic Regression model on a binary data with replications, the assumption is that the response variables(Yi) come from a normal distribution.
False - The assumption is that the response variable comes from a binomial distribution
The null hypothesis for the GOF of a Logistic Regression model is that the model does not fit the data.
False - The null hypothesis is the model fits the data
In regularized regression, the penalization is generally applied to all regression coefficient (Bo, ...., Bp) where p = number of predictors
False - The shrinkage penalty is applied to B1, ...., Bp but not to the intercept Bo
We can diagnose the constant variance assumption in Poisson regression using the normal probability plot.
False - There is not a constant variance assumption in Poisson Regression
If a Poisson regression model is found to be overdispersed, there is an indication that the variability of the response variable implied by the model is larger than the variability present in the observed response variable
False - This indicates the variability of the response variable implied by the model is smaller than the variability present in the observed response variable.
IN MLR, a VIF value of 6 for a predictor means that 92% of the variance in that predictor can be modeled by other predictors
False - Use VIF = 1/1 - Rj^2 to solve for Rj^2
Variable selection is a simple and solved statistical problem since we can implement it using the R software
False - Variable selection for a large number of predicting variables is an "unsolved" problem, and variable selection approaches should be tailored to the problem at hand.
If the constant variance assumption does not hold in MLR, we apply Box-Cox transformation to the predicting variables.
False - We apply a Box-Cox transformation to the response variable.
In stepwise regression, we accept/remove variables that produce larger AICs or BICs
False - We desire model that has the smallest AIC or BIC
When testing a subset of coefficients, deviance follows a Chi-square distribution with q Degree of Freedom, where q is the number of regression coefficients in the reduced model
False - q is number of regression coefficients discarded from the full model to get the reduced model
The estimated coefficients of a regression line is positive, when the coefficient of determination is positive.
False - r squared is always positive.
In multiple linear regression, as the value of R-squared increases, the relationship between predictors becomes stronger
False - r squared measures how much variability is explained by the model, NOT how strong the predictors are.
Ridge regression is a regularized regression approach that can be used for variable selection
False - regularized regression approached but it does not perform variable selection
In Logistic Regression, R2 can be used as a measure of explained variation in the response variable
False - response in logistic regression is binary. R2 is only used to explain variability in the dependent variable that is explained by the model.
Ridge regression cannot be used to deal with problems caused by high correlation among the predictors
False - ridge regression is used to deal with this problem actually
When dealing with a multiple linear regression model, an adjusted R-squared can be greater than the corresponding unadjusted R-Squared value.
False - the adjusted rsquared value take the number and types of predictors into account. It is lower than the r squared value.
The larger the number of variables in the model, the larger the training risk.
False - the larger the number of variables in a model the lower the training risk.
There is never a situation where a complex model is best.
False - there are situations where a complex model is best
In Poisson regression we model the error term
False - there is no error term
In logistic regression we have an additional error term to estimate.
False - there is not error term in logistic regression.
The estimators for the regression coefficients in the Poisson regression are biased.
False - they are unbiased
Generally models with covariance have high bias but low variance
False - they have low bias but high variance.
The variance estimator in logistic regression has a closed form expression.
False - use statistical software to obtain the variance-co-variance matrix
A t-test is used for testing the statistical significance of a coefficient given all predicting variables in a Poisson regression model.
False - use z-test for Poisson
For a linear regression under normality, the variance used in the Mallow's Cp penalty is the true variance, not an estimate of variance
False - variance used in Mallow's Cp penalty is the estimated variance from the full model
We can use residual analysis to conclusively determine the assumption of independence
False - we can only determine uncorrelated errors.
In logistic regression for goodness of fit, we can only use the Pearson residuals.
False - we can use Pearson or Deviance.
We cannot estimate the Regression coefficients of a MLR if predicting variables are linearly independent
False - we cannot predict if predictors are dependent (tricky part)
For Poisson regression we estimate the expectation of the log response variable.
False - we estimate the log of the expectation of the response variable.
In logistic regression we interpret the Betas in terms of the response variable.
False - we interpret it in terms of the odds of success or the log odds of success
If there is a high correlation between variables, Lasso will select both.
False lasso will select 1
A linear Regression model is a good fit to the data set if the Adjusted R2 is above 0.85
False- Adjust R2 is not a measure of GOF. GOF refers to all model assumptions holding
In regression inference, the 99% confidence interval of coefficient \beta_0 is always wider than the 95% confidence interval of \beta_1.
False- can only compare beta1 with beta1 and beta0 with beta0
. A correlation coefficient close to 1 is evidence of a cause-and-effect relationship between the two variables.
False- cause and effect can only be determined by a well designed experiment.
In simple linear regression models, we lose three degrees of freedom when estimating the variance because of the estimation of the three model parameters β 0 , β 1 , σ^2. (T/F)
False. See 1.2 Estimation Method "The estimator for σ 2 is σ ^ 2, and is the sum of the squared residuals, divided by n - 2." We lose two degrees of freedom because the variance estimator, σ ^ 2, uses only the estimates for β 0, and β 1 in its calculation.
With the Box-Cox transformation, when λ = 0 we do not transform the response. (T/F)
False. See 1.8 Diagnostics When λ = 0, we transform using the normal log.
In ANOVA, the linearity assumption is assessed using a plot of the response against the predicting variable. (T/F)
False. See 2.2 - Estimation Method Linearity is not an assumption of Anova.
What are the assumptions for multiple linear regression?
Linearity/Mean zero assumption, Constant Variance, Independence and Normality (for statistical inference)
In multiple linear regression, a VIF value of 6 for a predictor means that 90% of the variation in that predictor can be modeled by the other predictors. (T/F)
False. See Lesson 3.13: Model Evaluation and Multicollinearity A VIF value of 6 for a predictor means that 83.3% of the variation in that predictor can be modeled by the other predictors in the model.
Multicollinearity in multiple linear regression means that the rows in the design matrix are (nearly) linearly dependent. (T/F)
False. See Lesson 3.13: Model Evaluation and Multicollinearity Multicollinearity in multiple linear regression means that the columns in the design matrix are (nearly) linearly dependent.
Multicollinearity among the predicting variables will not impact the standard errors of the estimated regression coefficients. (T/F)
False. See Lesson 3.13: Multicollinearity Multicollinearity in the predicting variables can impact the standard errors of the estimated coefficients. "However, the bigger problem is that the standard errors will be artificially large."
In multiple linear regression, the prediction of the response variable and the estimation of the mean response have the same interpretation. (T/F)
False. See Lesson 3.2.9: Regression Line and Predicting a New Response. In multiple linear regression, the prediction of the response variable and the estimation of the mean response do not have the same interpretation.
A multiple linear regression model contains 6 quantitative predicting variables and an intercept. The number of parameters to estimate in this model is 7. (T/F)
False. See Lesson 3.2: Basic Concepts The number of parameters to estimate in a multiple linear regression model containing 6 quantitative predicting variables and an intercept is 8: 7 regression coefficients (β0,β1,...,β6) and the variance of the error terms (σ2).
Given a a quantitative predicting variable and a qualitative predicting variable with 7 categories in a linear regression model with intercept, 7 dummy variables need to be included in the model. (T/F)
False. See Lesson 3.2: Basic Concepts We only need 7 dummy variables. "When we have qualitative variables with k levels, we only include k-1 dummy variables if the regression model has an intercept."
The estimated variance of the error terms of a multiple linear regression model with intercept can be obtained by summing up the squared residuals and dividing the sum by n - p , where n is the sample size and p is the number of predictors. (T/F)
False. See Lesson 3.3: Regression Parameter Estimation The estimated variance of the error terms of a multiple linear regression model with intercept should be obtained by summing up the squared residuals and dividing that by n-p-1, where n is the sample size and p is the number of predictors as we lose p+1 degrees of freedom when we estimate the p coefficients and 1 intercept.
The causation of a predicting variable to the response variable can be captured using multiple linear regression on observational data, conditional of other predicting variables in the model. (T/F)
False. See Lesson 3.4 Model Interpretation "This is particularly prevalent in a context of making causal statements when the setup of the regression does not allow so. Causality statements can only be made in a controlled environment such as randomized trials or experiments. "
Conducting t-tests on each β parameter in a multiple linear regression model is the preferable to an F-test when testing the overall significance of the model. (T/F)
False. See Lesson 3.7: Testing for Subsets of Coefficients "We cannot and should not select the combination of predicting variables that most explains the variability in the response based on the t-tests for statistical significance because the statistical significance depends on what other variables are in the model."
Under testing a subset of coefficients, what is the distribution and degrees of freedom for the deviance?
For large sample size data, the distribution of this test statistic, assuming the null hypothesis is true, is a chi square distribution. With Q degrees of freedom where Q is the number of regression coefficients discarded from the full model to get the reduced model or the number of Z predicting variables.
Akaike Information Criterion
For linear regression under normality this is the training risk + penalty The complexity penalty is (2 * # of predictors in submodel * true_variance of full model)/n For AIC, we need to specify k = 2 Select the model with the smallest AIC
Using the Tukey method to find the confidence interval of the means, what does having a '0' in the CI mean?
For the confidence intervals that include zero, it's plausible that the difference between means is zero.
Forward Stepwise Regression
Forward stepwise regression starts with no predictors in the model and usually tends to select smaller models. As a result, this is preferable over backward stepwise regression. The same model is not necessarily the same as the one selected using backward stepwise regression
Which is preferred, forward or backward?
Forward, because it selects smaller models versus backwards which starts with a full model.
distribution of pearson residuals?
From the binomial approximation with a normal distribution using the central limit theorem
distribution of deviance residuals?
From the properties of the likelihood function, standard normal distribution if the model assumptions hold. That is, if the model is a good fit.
Multicollinearity
Generally occurs when there are high correlations between two or more predictor variables. In other words, one predictor variable can be used to predict the other. This creates redundant info, skewing the results in a regression model.
Combinations of Variables
Given p predicting variables, we can have 2^p combinations of the variables, and thus, 2^p models to choose from
When we are testing for overall regression for a Logistic model, what is the H0 and HA?
H0: all regression coefficients except intercept are 0 HA: at least one is not 0.
What are the null and alternative hypotheses of the F test?
H0: all the regression coefficients except the intercept are 0. HA: at least one is not 0.
Another GOF test is hypotheses testing, what is the H0 and HA?
H0: is that the model fits well. HA: the alternative is that the model does not fit well.
What is the null and alternative hypothesis for MLR?
H0: the coefficients are not0 HA: the coefficients (at least 1) are not equal to 0
H0 and HA for test of equal means?
H0: the means are equal HA: some means are different
What are the null and alternative hypotheses for ANOVA for MLR?
H0: the regression coefficients all equal zero. HA: At least one of the regression coefficients is not equal to zero.
The estimated versus predicted regression line for a given x*
Have the same expectation
The estimated versus predicted regression line for a given x*:
Have the same expectation
Forward-Backward Regression
Here we add and discard one variable at a time iteratively.
Generalized Linear Models (GLM)
Here, the response Y is assumed to have a distribution from the exponential family of distributions (Normal, Binomial, Poisson, Gamma etc) Under this model, we model a transformation g of the expectation of Y, given the predicting variables as a linear combination of the predicting variables We can write the expectation as the inverse of the g transformation of the linear combination of the predicting variables **Include table w/ link function & regression function pg 67
Hypothesis testing (statistical significance: +/-)
Here, the z-value is the same but the P-value will change Positive: P-value = P(Z > z-value) Negative: P-value = P(Z < z-value) **Applies for Logistic & Poisson Regression
Robust Regression
Here, we can estimate the regression coefficients in the presence of outliers When there are outliers in the distribution, it has heavy tails and we thus have departures from the normality assumption
Maximum Likelihood Estimate
Here, we maximize the likelihood function with respect to the model parameters or in this case, the regression coefficient The log likelihood that needs to be maximized is highly non-linear. Thus we cannot derive a closed form (exact) expression of the estimates. The estimated parameters and their standard errors are approximate estimates
high dimensionality
In linear regression, when the number of predicting variables P is large, we might get better predictions by omitting some of the predicting variables.
Nonparametric Regression
In non-parametric regression, the regression function has an unknown structure given the predicting variables, and the regression function does not depend on any parameters
Poisson Regression Assumptions
Linearity: The log transformation of the rate is a linear combination of the predicting variables. Independence: The response variables are independently obserterm-52ved Logit: The link function g is the log function. The log link function is almost always used
Logistic Regression Assumptions
Linearity: The relationship between the g of the probability of success and the predicted variable, is a linear function. Independence: The response binary variables are independently observed Logit: The logistic regression model assumes that the link function g is a logit function
An experiment was conducted to determine the effect of gamma radiation on the numbers of chromosomal abnormalities observed in cells. A multiple linear regression model was fitted to estimate the effect of the number of cells, amount of the radiation dose (Grays), and the rate of the radiation dose (Grays/hour) on the number of chromosomal abnormalities observed. The data frame has 27 observations. Here is the model summary and Cook's Distance plot. Coefficient Estimate SE t-value Pr(>|t|) (Intercept) -74.15392 42.24544 -1.755 0.092518 cells 0.06871 0.02196 3.129 0.004709** doseamt 41.33160 9.13907 4.523 0.000153*** doserate 20.28402 8.29071 2.447 0.022482* ---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 54.05 on X degrees of freedom Multiple R-squared: 0.5213, Adjusted R-squared: 0.4588 F-statistic: 8.348 on X and Y DF, p-value: 0.0006183 How does an increase in 1 unit in doserate affect the expected number of chromosome abnormalities, given that the other predictors in the model are held constant? Increase of 8.291 Decrease of 41.331 Increase of 20.284 Decrease of 9.134
Increase of 20.284 The estimated coefficient for doserate is 20.284. If we fix all other predictors, for each 1 unit increase in doserate, the expected number of chromosome abnormalities increases 20.284 units.
The assumption of normality:
It is needed for the sampling distribution of the estimators of the regression coefficients and hence for inference.
How do we interpret B1?
It is the estimated expected change in the response variable associated with one unit of change in the predicting variable.
What is the purpose of testing a subset of coefficients?
It simply compares two models and decides whether the larger model is statistically significantly better than the reduced model.
Weighted Least Squares (WLS)
It's a multiple regression model. But the difference is that we assume that the variance of the errors is not constant The vector of errors can be assumed to have a covariance-variance matrix sigma, thus allowing for correlated errors We have the independence assumption only when sigma matrix is a diagonal matrix
Penalties
L0: This is the number of nonzero regression coefficients. - Maximizing Q(betas) means searching through all submodels L1: This penalty applied to the vector of regression coefficients is equal to the sum of the absolute values of the regression coefficients to be penalized - Maximizing Q forces many betas to be zeros (Lasso) L2: This penalty applied to the vector of regression coefficients is equal to the sim of the squared regression coefficients to be penalized - Maximizing Q accounts for multicollinearity (Ridge)
LOOCV
LOOCV can be approximated by the sum between the training risk + the complexity penalty. The complexity penalty is (2 * # of predictors in submodel * estimated_variance of submodel)/n The variability of the submodel is smaller than that of the full model, thus LOOCV penalizes complexity less than Mallow's Cp LOOCV is ~ AIC when the true variance is replaced by the estimate of the variance from the submodel
Does the statistical inference for logistic regression rely on a small or large sample size?
Large, if it was a small then the statistical inference is not reliable
Lasso Regression
Lasso performs estimation of variable selections simultaneously The estimated regression coefficients from Lasso are less efficient than those provided by the ordinary least squares The penalty is the sum of absolute values of the regression coefficients except for intercept, that force coefficients to be zero We can apply lasso to standard linear regression, logistic regression, Poisson regression, and other linear models We do not have a closed-form expression for the estimated regression coefficients under this model Once a coefficient is non-zero, then it does not go back to zero In the case, where p the number of predictors is larger than n the number of observations, that is, more in the variables than observations, the Lasso selects, at most, n variables because of the nature of the convex optimization problem If there exists high correlation among predictors the prediction performance of the Lasso is dominated by ridge regression If there is a group of variables among which the correlation is very high, then the Lasso tends to select only one variable from that group Alpha = 1
3 assumptions of the logistic regression model
Linearity, Independence, Logit link function
what are the 4 assumptions of linear regression?
Linearity/Mean Zero, Constant Variance, Independence, Normality
Statistical Inference
Logistic Regression: Normal Distribution. The statistical inference based on the normal distribution applies only under large sample data. If the sample size, or n, is small? Then the statistical inference is not reliable i.e. warn on the lack of the reliability of the results Standard Regression: T-Distribution. The statistical inference relies on the distribution that applies under both small and large samples **Applies for Logistic & Poisson Regression
we estimate the Poisson model parameters using...
MLE
Introducing some bias yields a decrease in....
MSE
Mean Squared Error
MSE is commonly used to obtain estimators that may be biased, but less uncertain than unbiased ones. The MSE can be controlled It is possible to find a model with lower MSE than the unbiased/full model Introducing some bias yields a decrease in MSE followed by a later increase
4 types of Variable Selection Criteria
Mallow's Cp, AIC, BIC, LOOCV
Assumption of Independence
Means that the deviances, or in fact the response variables ys, are independently drawn from the data-generating process. (this most often occurs in time series data) This can result in very misleading assessments of the strength of regression.
Linearity/Mean zero assumption
Means that the expected value (deviances) of errors is zero. This leads to difficulties in estimating B0 and means that our model does not include a necessary systematic component
L2 penalty - Ridge
Minimizing the penalized least squares using this penalty accounts for multicollinearity, but does not perform variable selection. The result in regularized regression is a so-called ridge regression
The estimators of the linear regression model are derived by:
Minimizing the sum of squared differences between observed and expected values of the response variable.
What do we mean by model parameters in statistics?
Model parameters are unknown quantities, and they stay unknown regardless how much data are observed. We estimate those parameters given the model assumptions and the data, but through estimation, we're not identifying the true parameters. We're just estimating approximations of those parameters.
Poisson Regression
Model response data in the form of counts Modeling the expectation of the response variable as the exponential of the linear combination of the predicting variables Assumptions: - Log transformation of the rate is a linear combination of the predicting variables - The response count data are independently observed - The link function data is the log function The only model parameters are the regression coefficients estimated using the maximum likelihood approach Statistical inference on the regression coefficients using an approximation of the sampling distribution, the normal distribution
Variable Selection
Models with many predictors/covariates have low bias but high variance. Models with few predictors/covariates have high bias but low variance
Pooled variance estimator - N vs k?
N = combined size of samples, k = # of samples
If p(predictors) is large, is it feasible to fit a large number of submodels?
No
Is testing a subset of coefficients a GOF test?
No
Do the three stepwise approaches select the same model?
No, especially when p is large.
A data point far from the mean of the x's and y's is always: an influential point and an outlier a leverage point but not an outlier an outlier and a leverage point an outlier but not a leverage point None of the above
None of the above See 1.9 Outliers and Model Evaluation We only know that the data point is far from the mean of x's and y's. It only fits the definition of a leverage point because the only information we know is that it is far from the mean of the x's. So you can eliminate the answers that do not include a leverage point. That leaves us with remaining possibilities, "a leverage point but not an outlier" and "an outlier and a leverage point" , both of which we can eliminate. We do not have enough information to know if it is or is not an outlier . None of the answers above fit the criteria of it being always being a leverage point.
what is the distribution of B1?
Normal
Normal Distribution
Normal distribution relies on a large sample of data. Using this approximate normal distribution we can further derive confidence intervals. Since the distribution is normal, the confidence interval is the z-interval **Applies for Logistic & Poisson Regression
what can we use to check for normality?
QQ plot and histogram
T/F: An overdispersion parameter close to 1 indicates that the variability of the response is close to the variability estimated by the model.
T
What would we do if the T value is large?
Reject the null hypothesis that β1 is equal to zero. If the null hypothesis is rejected, we interpret this that β1 is statistically significant.
Time Series
Response data are correlated. This correlation results in a much smaller number of degrees of freedom than otherwise assumed under independence. Moreover, because of the correlation the data are concentrated into a smaller part of the probability space where the data lie. Ignoring dependence leads to inefficient estimates of regression parameters and leads to poor predictions. Standard errors are unrealistically small. i.e. too narrow confidence intervals thus improper statistical inferences
What are the two most common approaches to regularized regression?
Ridge and LASSO
For the G-o-F tests, do we reject the null hypotheses when the p-value is SMALL or BIG?
Small and we conclude the model is not a good fit
Cross validation
Split the data into two parts, first part called the training data and testing/validation data. The training data will be used to fit the model and thus get the estimated regression coefficients. The testing or validation data will be used to predict or classify the responses for this portion of the data, then compare to the observed response to estimate the classification error, one can repeat the process several times.
Poisson Regression vs Log transformed Linear Regression
Standard Regression: We estimate the expectation of the log of the response - E(log(Y)) The variance under the standard regression is assumed constant Poisson Regression: We estimate the log of the expectation of the response - log(E(Y)) The variance of the response is assumed to be equal to the expectation; thus, the variance is not constant. **Use the Poisson regression especially when the response data are small counts **Using the standard linear regression with log transformation instead of Poisson regression, will result in violations of the assumption of constant variance **Standard Linear Regression could be used if the number of counts are large and with the variance stabilizing transformation √(µ + 3/8) i.e. Square root of the response + 3/8. This transformation will work well for large count when the response data are large counts
Overall Regression
Standard Regression: We use the F test to test for the overall regression Logistic Regression: We use the difference between the log likelihood function of the model under the null hypothesis (also called the null-deviance), and the log likelihood of the full model (residual deviance) i.e. the difference between the null deviance and the residual deviance
T/F: Although there are no error terms in a logistic regression model using binary data with replications, we can still perform residual analysis.
T
T/F: An approximate test can be used to test for the overall regression in Poisson regression.
T
Assumptions - Mixed Effects
The error term is normally distributed with zero mean and constant variance The group effect is normally distributed with mu 0 and constant variance, where the variances of the error terms and of the group effect are two different parameters In random effects model, the observations are no longer independent, even if the error terms are independent.
What is the partial F-test?
The hypothesis test for whether a subset of regression coefficients are all equal to zero.
Coefficient Test (Deviance)
The hypothesis testing procedure is testing the null hypothesis that all alpha coefficients are zero, versus the alternative that at least one alpha coefficient is not zero For the testing procedure for subsets of coefficients, we compare the likelihood of a reduced model versus a full model. This test provides inferences on the predictive power of the model. Predictive power means that the predicting variables predict the data even if one or more of the assumptions do not hold
Logit link function assumption
The logistic regression model assumes that the link function is a so-called logit function. This is an assumption since the logit function is not the only function that yields s-shaped curves. And it would seem that there is no reason to prefer the logit to other possible choices.
Log odds function
The logit function which is the log of the ratio between the probability of a success and the probability of a failure
Model parameters
The model parameters are the regression coefficients. There is no additional parameter to model the variance since there's no error term. For P predictors, we have P + 1 regression coefficients for a model with intercept (beta 0). We estimate the model parameters using the maximum likelihood estimation approach
Linear Regression
The predicting variable is assumed to be fixed, whereas the response variable y is random We model their relationship as a linear function in x plus an error term epsilon Assumptions Linearity: Relationship between the response and the predicting variable is linear or the expectation of the error term is 0 Constant Variance: The variance of the error term is the same across all observations Independence: Uncorrelated errors Normality: Normality of the error terms We assume that the error terms are independent and identically distributed from a normal distribution with mean 0 and variance sigma squared The parameters are unknown but estimated based on the observed data Statistical inference in the regression coefficients is performed using the sampling distribution of the estimated regression coefficients. The t distribution with n-2 degrees of freedom
Regression Coefficient
The regression coefficient is interpreted as the log ratio of the rate with an increase with one unit in the predicting variable For p predictors, we have p + 1 regression coefficients for a model with intercept. We estimate the model parameters using the maximum likelihood estimation
Poisson Regression
The response Y in Poisson regression is assumed to have a Poisson distribution, and this is commonly used for modeling count or rate data. We assume that the i-th response Yi has a Poisson distribution, with rate lambda i. Alternatively, log of the rate lambda i is equal to the linear combination of the predicting variables We do not interpret beta with respect to the response variable but with respect to the ratio of the rate There is no error term
Generalized Linear Models
The response Y is assumed to have a distribution from the exponential family of distributions. Example of distributions in the exponential family of distributions are normal, binomial, Poisson, gamma We model a transformation g of the expectation of Y as a linear combination of the predicting variables The transformation g is called the link function, since it links the expectation of the response data to the predicting variables. The transformations g depends on the distribution of the response data
Overall Regression (Logistic)
The test statistic is a chi-squared distribution with p degrees of freedom where p is the number of predicting variables. We reject the null hypothesis when the P-value is small, indicating that the overall regression has explanatory power.
G transformation
The transformation g is called a link function since it links the expectation of the response to the predicting variables
The variability in the prediction comes from
The variability due to a new measurement and due to estimation.
The variability in the prediction comes from:
The variability due to a new measurement and due to estimation.
The mean squared errors (MSE) measures:
The within-treatment variability.
Wald Test (Z-test)
The z-test value is the ratio between the estimated coefficient minus 0, (which is the null value) divided by the standard error We reject the null hypothesis that the regression coefficient is 0 if the z value (gets too large) is larger in absolute value than the z critical point, (or the 1- alpha over 2 of the normal quantile). We interpret that the coefficient is statistically significant **Applies for Logistic & Poisson Regression
Reasons why a model may not be a good fit
There may be other variables that should be included in the model The relationship between Logit of the expected probability and predictors might be multiplicative, rather than additive Departure from the linearity assumption Initial observations outliers, leverage points are also still an issue for this model Logit function does not fit well with the data The binomial distribution isn't appropriate. For example, if there's correlation among the responses or there's heterogeneity in the success that hasn't been modeled. Both of these violations can lead to what we call overdispersion
Nonlinear & Nonparametric Regression
These approaches are commonly used to deal with nonlinearity
Regularized (Penalized) Regression
These approaches perform variable selection and estimation simultaneously.
Deviance Residuals
These are the signed square root of the log-likelihood evaluated at the saturated model when we assume that the estimate expected response is the observed response versus the fitted model Deviance residuals have a standard normal distribution if the model is a good fit (i.e. model assumptions hold)
Spatial Regression
This approach also deals with correlated data If the assumption of uncorrelated errors does not hold, it can lead to misleading assessments of the strength of the regression Spatial processes can be observed over regular grid such as images. A regular grid is also called the lattice. Spatial processes can also be observed on irregular grids
Mixed Effects Models
This approach deals with replications in the response data Generally, a group effect is random if we can think of the responses we observe in that group to be samples from a larger population
Mallows Cp
This approach is useful when there are no control variables. It assumes we can estimate the variance from the full model. **This is not the case when p > n The complexity penalty is (2 * # of predictors in submodel * estimated_variance of full model)/n Select the model with the smallest Cp score
Type I Error
This happens if the sample size, or n is small. The hypothesis testing procedure will have a probability of Type I error larger than the significance level (i.e. more Type I errors than expected) **Applies for Logistic & Poisson Regression
Cross Validation
This is a direct measure of predictive power Random sampling is computationally more expensive than the K-fold cross validation, with no clear advantage in terms of the accuracy of the estimation classification error rate The rule of thumb for choosing K is about K = 10 LOOCV is a K-fold cross validation with K = n. The larger K is, the larger the number of folds, the less bias the estimate of the classification the error is but has higher variability.
Leave-One-Out Cross Validation (LOOCV)
This is a direct measure of predictive power. This is just like Mallow's, except the variance is for the S submodel, not the full model. The LOOCV penalizes complexity less than Mallow's Cp.
Stepwise Regression
This is a heuristic search used when p is large. It's also useful when there are control variables. The three stepwise regression approaches do not necessarily select the same model, especially when p is large This is a greedy algorithm; it does not guarantee to find the model with the best score Greedy means we always take the biggest jump (up or down) in the selected criterion
General Additive Regression
This is a non-parametric model used if the linearity assumption does not hold and/or it's difficult to identify transformations that improve the fit Here the relationship of predicting variable to the response is assumed unknown The estimation of the parameters does not have a closed form expression
Normality assumption
This is needed if we want to do any confidence or prediction intervals, or hypothesis test, which we usually do. If this assumption is violated, hypothesis test and confidence and prediction intervals and be very misleading.
Deviance
This is the difference between the log likelihood from a reduced model and the log likelihood from a full model For large sample size data, the distribution (assuming the null hypothesis is true), is a chi square distribution with Q degrees of freedom Q = Number of Z predicting variables (controlling variables for bias selection) i.e. the number of regression coefficients discarded from the full model to get the reduced model The P-value of the test is computed as the right tail of the chi-square distribution with Q degrees of freedom of the test value (Deviance) **This test is NOT a goodness of fit test. It simply compares two models and decides whether the larger model is statistically significantly better than the reduced model.
Odds of a success
This is the exponential of the Logit function
Prediction Risk
This is the measure of the bias-variance tradeoff It is the sum of expected squared differences between fitted values by the model S and future observations Prediction Risk = variance(future observation) + bias^2 + variance(prediction) MSE = bias^2 + variance(prediction) The variance of the future observation (sigma squared) is an irreducible error and thus cannot be controlled
Logistic Regression
This is the model where the link function g is the logit function. The link function g is the log of the ration of p and 1-p We model the probability of a success given the predicting variable using the g link function in such a way that the g function of the probability of the success is a linear model of the predicting variables. The g function is the S-shaped function that models the probability of success with respect to the predicting variables. Assumptions: - Linearity in the predicting variables - Independence in the response of the observed data - The link function is the logit function The parameters for logistic regression are beta 0, beta 1 and beta p. These parameters are unknown but estimated based on the observed data using the maximum likelihood approach Statistical inference in the regression coefficient using an approximate sampling distribution of the estimated regression coefficients, where the approximate distribution is the normal distribution
Exposure of the response variable
This is the number of units when modeling rate data with Poisson Regression Exposure is accounted for using an offset which is the log of the exposure. This variable is added as a predictor to the Poisson regression model
Mallow's Cp
This is the oldest approach to variable selection. This assumes that we can estimate the variance from the full model, however this is NOT the case when p is larger than n.
Training Error
This is the proportion of the responses that are misclassified We cannot use the training error rate as an estimate of the true error classification error rate because it is biased downward The bias comes from the fact that we use the data twice. One, we used it for fitting the model and the second time is to estimate the classification error rate
Pearson Residuals
This is the standardized difference between the ith observed response and estimated expected response, which is ni times the probability of success We need to standardize the difference between observed and expected response, as the responses have different variances Pearson residuals have an approximately standard normal distribution
Training Risk
This is the sum of squared differences between fitted values for sub model S and the observed values. The training risk is biased upward because the prediction risk is used twice The larger the number of variables in the model, the larger the training risk is
Simpson's paradox
This is when the addition of a predictive variable reverses the sign on the coefficients of an existing parameter It refers to reversal of an association when looking at a marginal relationship versus a partial or conditional one. This is a situation where the marginal relationship adds a wrong sign This happens when the 2 variables are correlated
Complete Separation
This is when the model fits perfectly (p-value = 1) after an outlier is removed It indicates that the possibility of a simpler model being good enough should be explored
Nonlinear Regression
This is when the relationship between the response variable and the predicting variables is known but it cannot be expressed as a sum of transformed predicting variables. The regression function has a known structure given the predicting variables. It depends on a series of parameters In nonlinear regression, we know the function F except for the theta. We estimate the model by minimizing the sum of squared errors. We minimize this with respect to theta
Overdispersion
This is where the variability of the probability estimates is larger than would be implied by a binomial random variable ɸ = D/(n-p-1) D is the Deviance(sum of squared deviances) If ɸ > 2 then model is overdispersed; an over-dispersed model will fit better Overdispersion impacts the estimated variance and statistical inference. If overdispersion is not accounted for, statistical inference will not be as reliable
You were hired to consult on a study for the attendance behavior of high school students at two different schools. The data set you were given contains for each 316 students: the number of days he/she was absent in an academic year (daysabs), his/her math scores (math), his/her language arts scores (langarts), and whether the student is male or female (1 = male, 0 = female). A Poisson regression model was fitted to evaluate the relationship between the number of days of absence in an academic year and all the predictors. The R output for the model summary is as follows: Coefficient Estimate SE z value Pr(>|z|) (Intercept) 2.687666 0.072651 36.994 <2e-16 math -0.003523 0.001821 -1.934 0.0531 langarts -0.012152 0.001835 -6.623 3.52e-11 male -0.400921 0.048412 -8.281 <2e-16 Also, assume the average language arts scores (across all students) is 50, and the average math scores (across all students) is 45.5. How many regression coefficients including the intercept are statistically significant at the significance level 0.05? All Three Two None
Three As the summary output above shows, the coefficients associated to the intercept, langarts and male are statistically significant at α = 0.05. Their associated p-values (<2e-16, 3.52e-11, <2e-16) are smaller than 0.05
Hypothesis Testing (coefficient == 0)
To perform hypothesis testing, we can use the approximate normal sampling distribution. The resulting hypothesis test is also called the Wald test since it relies on the large sample normal approximation of MLEs To test whether the coefficient betaj = 0 or not, we can use the z- value **Applies for Logistic & Poisson Regression
Hypothesis Testing (coefficient == constant)
To test if the regression coefficient is equal to this constant b, then the z-value changes. We subtract b from the estimated coefficients of the numerator We decide to reject/accept using the P-value The P-value is 2 times the left tail of the standard normal of the quantile provided by the absolute value of the z-value P-value = 2P(Z > |z-value|) **Applies for Logistic & Poisson Regression
Alpha is only used in elastic net.
True
Time Series Regression Characteristics
Trends: Can be long-term increase/decrease in the data over time or it can fluctuate mostly over time Seasonality: This is influenced by seasonal factors. Periodicity: This is when seasonality repeats exactly at the same time with exactly the same regular pattern Cyclical Trends: This is when data exhibits rises and falls that are not of a fixed period Heteroskedasticity: This means the variability varies with time Correlation: Successive observations are similar or negative/positive To account for trend and seasonality or periodicity, we recompose a time series into three components, mt (trend), st(seasonality), and Xt (stationary process). In other words its probability distribution does not change when shifted in time
1. The means of the k populations 2. The sample means of the k populations 3. The sample means of the k samples are NOT all the model parameters in ANOVA
True
A MLR model has high explanatory power if the coefficient of determination is close to 1
True
A high Cook's distance for a particular observation suggests that the observation could be an influential point.
True
A logistic regression model may not be a good fit to the data if the responses are correlated or if there is heterogeneity in the success that hasn't been modeled
True
A negative value of LaTeX: \beta_1 β 1 is consistent with an inverse relationship between LaTeX: x x and LaTeX: y y .
True
AIC looks just like the Mallow's Cp except that the variance is the true variance and not its estimate.
True
In a full model F test, a low p-value indicates the model has predictive power.
True
In a multiple regression model with 7 predicting variables, the sampling distribution of the estimated variance of the error terms is a chi-squared distribution with n-8 degrees of freedom.
True
In case of multiple linear regression, controlling variables are used to control for sample bias.
True
In evaluating a multiple linear model residual analysis is used for goodness of fit assessment.
True
In evaluating a multiple linear model the F test is used to evaluate the overall regression.
True
In evaluating a multiple linear model the coefficient of variation is interpreted as the percentage of variability in the response variable explained by the model.
True
In evaluating a multiple linear model, Residual analysis is used for goodness of fit assessment.
True
In evaluating a multiple linear model, the F test is used to evaluate the overall regression.
True
In evaluating a multiple linear model, the coefficient of variation is interpreted as the percentage of variability in the response variable explained by the model.
True
In evaluating a simple linear model residual analysis is used for goodness of fit assessment.
True
In evaluating a simple linear model the coefficient of variation is interpreted as the percentage of variability in the response variable explained by the model.
True
In evaluating a simple linear model there is a direct relationship between coefficient of variation and the correlation between the predicting and response variables.
True
In multiple linear regression, controlling variables are used to control for sample bias.
True
In simple linear regression, we can diagnose the assumption of constant-variance by plotting the residuals against fitted values.
True
In testing for a subset of coefficients in logistic regression the null hypothesis is that the coefficient is equal to zero
True
It is possible to apply logistic regression when the response variable Y has 3 classes.
True
It is possible to produce a model where the overall F-statistic is significant but all the regression coefficients have insignificant t-statistics.
True
Let LaTeX: Y^* Y ∗ be the predicted response at LaTeX: x^* x ∗ . The variance of LaTeX: Y^* Y ∗ given LaTeX: x^* x ∗ depends on both the value of LaTeX: x^* x ∗ and the design matrix.
True
Under the normality assumption, the estimator for β 1 is a linear combination of normally distributed random variables. (T/F)
True See 1.4 Statistical Inference "Under the normality assumption, β 1 is thus a linear combination of normally distributed random variables... β ^ 0 is also linear combination of random variables"
If the model assumptions hold, then the estimator for the variance, σ ^ 2, is a random variable. (T/F)
True See 1.8 Statistical Inference We assume that the error terms are independent random variables. Therefore, the residuals are independent random variables. Since σ ^ 2 is a combination of the residuals, it is also a random variable.
An ANOVA model with a single qualitative predicting variable containing k groups will have k + 1 parameters to estimate. (T/F)
True See 2.2 Estimation Method We have to estimate the means of the k groups and the pooled variance estimator, s pooled ^2.
If the constant variance assumption in ANOVA does not hold, the inference on the equality of the means will not be reliable. (T/F)
True See 2.8 Data Example "This is important since without a good fit, we cannot rely on the statistical inference." Only when the model is a good fit, i.e. all model assumptions hold, can we rely on the statistical inference.
If the pairwise comparison interval between groups in an ANOVA model includes zero, we conclude that the two means are plausibly equal. (T/F)
True See 2.8 Data Example If the comparison interval includes zero, then the two means are not statistically significantly different, and are thus, plausibly equal.
The logit function is the log of the ratio of the probability of success to the probability of failure. It is also known as the log odds function.
True The logit link function is also known as the log odds function.
In logistic regression, the relationship between the probability of success and the predicting variables is nonlinear.
True We model the probability of success given the predictors by linking the probability to the predicting variables through a nonlinear link function.
Visual Analytics for logistic regression Normal probability plot of residuals Residuals vs predictors Logit of success rate vs predictors
True Normal probability plot of residuals - Normality Residuals vs predictors - Linearity/Independence Logit of success rate vs predictors - Linearity
Elastic net regression uses both penalties of ridge and lasso regression and hence combines the benefits of both
True -
For a classification model, the training error rate tends to underestimate the true classification error rate of the model.
True -
In the balance of Bias-Variance tradeoff, adding variables to our model tends to increase our variance and decrease our bias
True - Adding more variables will increase the variability and possibly induce multicollinearity. Adding more variables also reduces the bias in the model since it has an additional predictor to conform to which keeps the model from favoring one of the original predictors.
BIC variable selection criteria favors simpler models
True - BIC penalize complexity more than other approaches.
When the objective is to explain the relationship to the response, one might consider including predicting variables which are correlated
True - But this should be avoided for prediction
Ridge regression corrects for the impact of multicollinearity by reweighting the regression coefficients
True - Ridge regression has been developed to correct for the impact of multicollinearity. If there is multicollinearity in the model, all predicting variables are considered to be included in the model but ridge regression will allow for re-weighting the regression coefficients in a way that those corresponding to correlated predictor variables share their explanatory power and thus minimizing the impact of multicollinearity on the estimation and statistical inference of the regression coefficients.
We can estimate the regression coefficients in Poisson regression using the maximum likelihood estimation approach
True - Use MLE for Poisson
To perform hypothesis testing for Poisson, we can use again the approximate normal sampling distribution, also called the Wald test
True - Wald Test also used with logistic regression
Although there are no error terms in a Logistic Regression model using binary data with replications, we can still perform residual analysis.
True - We can perform residual analysis on the Pearson residual or the Deviance residual.
When selecting variables for explanatory purpose, one might consider including predicting variables which are correlated if it would help answer your research hypothesis
True - When the objective is to explain the relationship to the response, one might consider including the predicting variables even they are correlated
Under the null hypothesis of good fit for logistic regression, the test statistic has a Chi-Square distribution with n- p- 1 degrees of freedom
True - don't forget, we want large P values
A Poisson regression model fit to a dataset with a small sample size will have a hypothesis testing procedure with more Type I errors than expected
True - if sample size is small, the statistical inference is not reliable. Thus, the hypothesis testing procedure will have a probability of type I error larger than the significant level
L1 penalty will force many betas, many regression coefficients to be 0s
True - is equal to the sum of the absolute values of the regression coefficients to be penalized
L2 does not perform variable selection
True - is equal to the sum of the squared regression coefficients to be penalized and does not do variable selection
Stepwise regression is a greedy search algorithm that is not guaranteed to find the model with the best score
True - it does not guarantee to find the model with the best score
The assumptions in logistic regression are - Linearity, Independence of response variable, and the link function is the logit function.
True - linearity is measured through the link, , the g of the probability of success and the predicted variable.
The prediction interval of one member of the population will always be larger than the confidence interval of the mean response for all members of the population when using the same predicting values. (T/F)
True. See 1.7 Regression Line: Estimation & Prediction Examples "Just to wrap up the comparison, the confidence intervals under estimation are narrower than the prediction intervals because the prediction intervals have additional variance from the variation of a new measurement."
The normality assumption states that the response variable is normally distributed. (T/F)
True. See 1.8 Diagnostics "Normality assumption: the error terms are normally distributed." The response may or may not be normally distributed, but the error terms are assumed to be normally distributed.
It is good practice to create a multiple linear regression model using a linearly dependent set of predictor variables. (T/F)
True. See Lesson 3.13: Model Evaluation and Multicollinearity It is good practice to create a multiple linear regression model using a linearly independent set of predicting variables. "XTX is not invertible if the columns of X are linearly dependent, i.e. one predicting variable, corresponding to one column, is a linear combination of the others."
If the residuals are not normally distributed, we can model the transformed response variable instead, where a common transformation for normality is the Box-Cox transformation. (T/F)
True. See Lesson 3.3.11: Assumptions and Diagnostics If the normality assumption does not hold, we can use a transformation that normalizes the response variable such as Box-Cox transformation.
A linear regression model has high explanatory power if the coefficient of determination is close to 1. (T/F)
True. See Lesson 3.3.13: Model Evaluation and Multicollinearity If R2 is close to 1, almost all of the variability in Y can be explained by the linear regression model; hence, the model has high explanatory power.
For a given predicting variable, the corresponding estimated regression coefficient will likely be different in a conditional model versus a marginal model. (T/F)
True. See Lesson 3.4: Model Interpretation "Importantly, the estimated regression coefficients for the conditional and marginal relationships can be different, not only in magnitude but also in sign or direction of the relationship."
In multiple linear regression, the estimated regression coefficient corresponding to a quantitative predicting variable is interpreted as the estimated expected change in the response variable when there is a change of one unit in the corresponding predicting variable holding all other predictors fixed. (T/F)
True. See Lesson 3.4: Model Interpretation "The estimated value for one of the regression coefficient βi represents the estimated expected change in y associated with one unit of change in the corresponding predicting variable, Xi, holding all else in the model fixed."
A partial F-Test can be used to test whether the regression coefficients associated with a subset of the predicting variables in a multiple linear regression model are all equal to zero. (T/F)
True. See Lesson 3.7: Testing for Subsets of Regression Parameters We use the Partial F-test to test the null hypothesis that the regression coefficients associated to a subset of the predicting variables are all equal to zero. The alternative hypothesis is that at least one of these regression coefficients is not zero.
Goodness of Fit (Poisson)
Use the Pearson or deviance residuals to evaluate whether they are normally distributed, if they are normally distributed then we conclude that the model is a good fit In a goodness of fit test, the null hypothesis is that the model fits well and the alternative that the model does not fit well. The test statistic for the goodness of fit test is the sum of square root deviances Chi-squared distribution of n-p-1 degrees of freedom If the p-value is small. We reject the null hypotheses of good fit and thus, we conclude that the model is not a good fit
Goodness of Fit (binary data with no responses)
Use the deviances from the aggregated model for goodness of fit, not based on the individual level data
L2 Penalty
Using this penalty accounts for multicollinearity, but does not perform variable selection. The resulting regularized regression is called the Ridge regression This is easy to implement, but it does not measure sparsity not perform variable selection
L1 Penalty
Using this penalty will force many betas to be 0s. The resulting regularized regression is called the LASSO regression L1 penalty measures sparsity
L0 Penalty
Using this penalty, the penalized least squares is equivalent to searching over all models and thus not feasible for a large number of predictive variables L0 penalty provides the best model given selection criteria, but it requires fitting all submodels
How do we interpret the VIF?
VIF measures the proportional increase in the variance of beta hat j compared to what it would have been if the predictive variables had been completely uncorrelated
Uncorrelated Errors Assumption
Violations of this assumption can lead to misleading assessment of the strength of the regression. This is because the degrees of freedom are not equal to the sample size. In fact, there are less degrees of freedom due to the correlation. Moreover, not accounting for correlation will result in higher variability or uncertainty in the estimate, thus less reliable statistical inference
Goodness of Fit
We can use the Pearson or Deviance residuals to evaluate whether they are normally distributed. If they're normally distributed, we conclude that the model is a good fit If the model is not a good fit, it means the linearity assumption may not hold
Logistic Regression Coefficient
We interpret the regression coefficient beta as the log of the odds ratio for an increase of one unit in the predicting variable We do not interpret beta with respect to the response variable but with respect to the odds of success The estimators for the regression coefficients in logistic regression are unbiased and thus the mean of the approximate normal distribution is beta. The variance of the estimator does not have a closed form expression
g-function
We link the probability of success to the predicting variables using the g link function. The g function is the s-shape function that models the probability of success with respect to the predicting variables The link function g is the log of the ratio of p over one minus p, where p again is the probability of success Logit function (log odds function) of the probability of success is a linear model in the predicting variables The probability of success is equal to the ratio between the exponential of the linear combination of the predicting variables over 1 plus this same exponential
What can we do since the training risk is biased?
We need to correct for this bias by penalizing the training risk by adding a complexity penalty.
To use the estimated residuals for assessing model assumptions, what do we need to do first?
We need to standardize them
When would we reject the null hypothesis for a z test?
We reject the null hypothesis that the regression coefficient is 0 if the z value is larger in absolute value than the z critical point. Or the 1- alpha over 2 normal quanta. We interpret this that the coefficient is statistically significant.
Regression Coefficients - Robust Regression
We replace the sum of squared errors with the sum of absolute errors, thus estimating the regression coefficients by minimizing the sum of absolute errors. We cannot obtain closed or exact expressions for the estimated regression coefficients. The estimated coefficients are approximate estimates
We detect departure from the assumption of constant variance
When the residuals vs fitted values are larger in the ends but smaller in the middle.
Robust Regression II
When we have a heavy tailed distribution that has symmetry, we need to replace the normal distribution with the PDF The main difference between this distribution and that of a normal distribution is that we have the absolute value of the difference between y and the centrality parameter mu whereas for normal distribution, we had y minus mu squared. This distribution has heavier tails than the normal The parameter mu is not the mean or the expectation anymore, but it is the median of a distribution. The estimated mu using this approach is the sample median
High Dimensionality
When we have a very large number of predicting variables to consider, it can be difficult to interpret and work with the fitted model
training risk
compute the prediction risk for the observed data and take the sum of squared differences between fitted values for sub model S and the observed values
In a study of cheddar cheese from the LaTrobe Valley of Victoria, Australia, samples of cheese were analyzed for their chemical composition and were subjected to taste tests. Overall taste scores were obtained by combining the scores from several tasters. The data frame has 30 observations and the following variables: taste - a subjective taste score Acetic - concentration of acetic acid (log scale) H2S - concentration of hydrogen sulfide (log scale) Lactic - concentration of lactic acid Using the following R output from a fitted multiple linear regression model, answer the following multiple-choice questions. Call: lm(formula = taste ~., data = chedder) Residuals: Min 1Q Median 3Q Max -17.390 -6.612 -1.009 4.908 25.449 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -28.8768 19.7354 -1.463 0.15540 Acetic 0.3277 4.4598 0.073 0.94198 H2S 3.9118 1.2484 3.133 0.00425 ** Lactic 19.6705 8.6291 2.280 0.03108 * --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' Residual standard error: 10.13 on 26 degrees of freedom Multiple R-squared: 0.6518, Adjusted R-squared: 0.6116 F-statistic: 16.22 on 3 and 26 DF, p-value: 3.81e-06 Calculate the sum of squared errors (SSE) from the given R output. Select the choice that most closely approximates your calculation. a. 102.617 b. 2668.039 c. 2533.081 d. 2786.025
b. 2668.039 MSE = SSE/(n−p−1) = SSE/DF. Hence, SSE =MSE*DF = 10.132* (30-3-1) = 2668.039
A linear regression model was fitted to estimate the response variable Height for black cherry trees using just the Diameter. The data frame has 31 observations. Here is the model summary, with some parts missing. Coefficients: Estimate Std. Error t-value Pr(>|t|) (Intercept) 62.0313 A 14.152 1.49e-14 *** Diameter 1.0544 0.3222 3.272 0.00276 ** --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 5.538 on B degrees of freedom Multiple R-squared: 0.2697, Adjusted R-squared: 0.2445 What is the value of A (standard error for the estimated intercept)? a. 877.9 b. 4.383 c. 0.2281 d. None of the above
b. 4.383 Since t-value = (estimated intercept - 0)/estimated std, we have, estimated std = estimated intercept/tvalue = 62.0313/14.152 = 4.383
Which of the following is not an application of regression? a. Testing hypotheses b. Proving causation c. Predicting outcomes d. Modeling data
b. Proving causation
We can make causality statements for...
experimental designs
Why can we not use the training error rate as an estimate of the true error classification error rate?
because it is biased downward. And the bias comes from the fact that we use the data twice. First, we used it for fitting the model and the second time to estimate the classification error rate.
In logistic regression, how do we define residuals for evaluating g-o-f?
binary data with replications.
What is the distribution of binary data WITH replications?
binomial distribution with more than one trial or ni greater than 1
In a study of cheddar cheese from the LaTrobe Valley of Victoria, Australia, samples of cheese were analyzed for their chemical composition and were subjected to taste tests. Overall taste scores were obtained by combining the scores from several tasters. The data frame has 30 observations and the following variables: taste - a subjective taste score Acetic - concentration of acetic acid (log scale) H2S - concentration of hydrogen sulfide (log scale) Lactic - concentration of lactic acid Using the following R output from a fitted multiple linear regression model, answer the following multiple-choice questions. Call: lm(formula = taste ~., data = chedder) Residuals: Min 1Q Median 3Q Max -17.390 -6.612 -1.009 4.908 25.449 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -28.8768 19.7354 -1.463 0.15540 Acetic 0.3277 4.4598 0.073 0.94198 H2S 3.9118 1.2484 3.133 0.00425 ** Lactic 19.6705 8.6291 2.280 0.03108 * --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' Residual standard error: 10.13 on 26 degrees of freedom Multiple R-squared: 0.6518, Adjusted R-squared: 0.6116 F-statistic: 16.22 on 3 and 26 DF, p-value: 3.81e-06 Calculate the sum of squares total (SST) from the given R output. Select the choice that most closely approximates your calculation. a. 4994.48 b. 3147.54 c. 7662.38 d. 8655.21
c. 7662.38 Since, R2 = 1 - SSE/SST, we have, SST = SSE /(1-R 2 ) = 2668.039/(1-0.6518) = 7662.38
You have measured the systolic blood pressure of a random sample of 50 employees of a company, and have fitted a linear regression model to estimate the response variable systolic blood pressure using the sex of the employees. The 95% confidence interval for the mean systolic blood pressure for the female employees is computed to be (122, 138). Which of the following statements gives a valid frequentist interpretation of this interval? a. 95% of the sample of female employees has a systolic blood pressure between 122 and 138. b. 95 % of the employees in the company have a systolic blood pressure between 122 and 138. c. If the sampling procedure were repeated 100 times, then approximately 95 of the resulting 100 confidence intervals would contain the true mean systolic blood pressure for all female employees of the company. d. We are 95% confident the sample mean is between 122 and 138
c. If the sampling procedure were repeated 100 times, then approximately 95 of the resulting 100 confidence intervals would contain the true mean systolic blood pressure for all female employees of the company.
In a study of cheddar cheese from the LaTrobe Valley of Victoria, Australia, samples of cheese were analyzed for their chemical composition and were subjected to taste tests. Overall taste scores were obtained by combining the scores from several tasters. The data frame has 30 observations and the following variables: taste - a subjective taste score Acetic - concentration of acetic acid (log scale) H2S - concentration of hydrogen sulfide (log scale) Lactic - concentration of lactic acid Using the following R output from a fitted multiple linear regression model, answer the following multiple-choice questions. Call: lm(formula = taste ~., data = chedder) Residuals: Min 1Q Median 3Q Max -17.390 -6.612 -1.009 4.908 25.449 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -28.8768 19.7354 -1.463 0.15540 Acetic 0.3277 4.4598 0.073 0.94198 H2S 3.9118 1.2484 3.133 0.00425 ** Lactic 19.6705 8.6291 2.280 0.03108 * --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' Residual standard error: 10.13 on 26 degrees of freedom Multiple R-squared: 0.6518, Adjusted R-squared: 0.6116 F-statistic: 16.22 on 3 and 26 DF, p-value: 3.81e-06 Given the R output, an increase in the concentration of lactic acid by one unit results in a(n) ___________ in the given taste score by ___________ points, holding all other variables constant. a. Decrease, 19.6705 b. Increase, 8.6291 c. Increase, 19.6705 d. Decrease, 8.6291
c. Increase, 19.6705 The estimated coefficient for Lactic is 19.6705. If we fix all other predictors, for each 1 unit increase in Lactic, the given test score increases 19.6705 points.
In ANOVA, for which of the following purposes is the Tukey method used? a. Test for homogeneity of variance b. Test for normality c. Test for differences in pairwise means d. Test for independence of errors
c. Test for differences in pairwise means
marginal model (SLR)
captures the association of one predicting variable to the response variable marginally, that means without consideration of other factors.
What is the sampling distribution for the pooled variance estimator?
chi-square distribution with N - K degrees of freedom
What is the estimated sampling distribution of s^2?
chi-square with n-1 DF
What is the estimated sampling distribution of sigma^2?
chi-square with n-2 DF (~ equivalent to MSE)
in MLR, the sampling distribution for sigma^2 is MSE .....
chi-square with n-p-1 DF
σ^2 hat distribution is?
chi-square, n-p-1 DF
What is the distribution and DOF of overall regression test statistic?
chi-squared with p degrees of freedom where p is the number of predicting variables.
Poisson regression
commonly used for modeling count or rate data.
A linear regression model was fitted to estimate the response variable Height for black cherry trees using just the Diameter. The data frame has 31 observations. Here is the model summary, with some parts missing. Coefficients: Estimate Std. Error t-value Pr(>|t|) (Intercept) 62.0313 A 14.152 1.49e-14 *** Diameter 1.0544 0.3222 3.272 0.00276 ** --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 5.538 on B degrees of freedom Multiple R-squared: 0.2697, Adjusted R-squared: 0.2445 What is the value of B (degrees of freedom of the estimated error variance)? a. 32 b. 31 c. 30 d. 29
d. 29 The degrees of freedom of the estimated error variance are calculated as df = n-k - 1 = 31 - 1 - 1 = 29
Models with few predictors have...
high bias but low variance
In GLM or generalized linear models, the response Y is assumed to have what kind of distribution?
distribution from the exponential family of distributions
L1 penalty - LASSO
equal to the sum of the absolute values of the regression coefficients to be penalized. Minimizing the penalized least squares using this penalty will force many betas, many regression coefficients to be 0s. (LASSO)
In a simple linear regression model, given a significance level α , the ( 1 − α ) % confidence interval for the mean response should be wider than the ( 1 − α ) % prediction interval for a new response at the predictor's value x∗ .
false In a simple linear regression model, given a significance level α, the (1−α)100% confidence interval for the mean response should be narrower than the (1−α)100% prediction interval for a new response at the predictor's value x* .
With k-fold cross validation larger k values increase bias and reduce variance.
false Larger values of k decrease bias and increase variance.
Stepwise regression is a greedy algorithm searching through all possible combinations of the predicting variables to find the model with the best score.
false Not all possible combinations are checked.
In a multiple linear regression model, when more predictors are added, R^2 can decrease if the added predictors are unrelated to the response variable.
false R^2 never decreases as more predictors are added to a multiple linear regression model.
Ridge regression is a regularized regression approach that can be used for variable selection.
false Ridge regression is a regularized regression approach but does not perform variable selection.
In simple linear regression models, we lose three degrees of freedom when estimating the variance because of the estimation of the three model parameters β 0 , β 1 , σ 2.
false See 1.2 Estimation Method "The estimator for σ 2 is σ ^ 2, and is the sum of the squared residuals, divided by n - 2."
The sampling distribution for the variance estimator in simple linear regression is χ 2 (chi-squared) regardless of the assumptions of the data.
false See 1.2 Estimation Method "The sampling distribution of the estimator of the variance is chi-squared, with n - 2 degrees of freedom (more on this in a moment). This is under the assumption of normality of the error terms."
We assess the constant variance assumption by plotting the error terms, ϵ i, against fitted values.
false See 1.2 Estimation Method "We use ϵ ^ i as proxies for the deviances or the error terms. We don't have the deviances because we don't have β 0 and β 1.
The simple linear regression coefficient, β ^ 0, is used to measure the linear relationship between the predicting and response variables.
false See 1.2 Estimation Method β ^ 0 is the intercept and does not tell us about the relationship between the predicting and response variables.
The p-value is a measure of the probability of rejecting the null hypothesis.
false See 1.5 Statistical Inference Data Example "p-value is a measure of how rejectable the null hypothesis is... It's not the probability of rejecting the null hypothesis, nor is it the probability that the null hypothesis is true."
The normality assumption states that the response variable is normally distributed.
false See 1.8 Diagnostics "Normality assumption: the error terms are normally distributed." The response may or may not be normally distributed, but the error terms are assumed to be normally distributed.
With the Box-Cox transformation, when λ = 0 we do not transform the response.
false See 1.8 Diagnostics When λ = 0, we transform using the normal log.
In ANOVA, the linearity assumption is assessed using a plot of the response against the predicting variable.
false See 2.2. Estimation Method Linearity is not an assumption of ANOVA.
For a multiple linear regression model to be a good fit, we need the linearity assumption to hold for only one of the predicting variables.
false See Lesson 3.11: Assumptions and diagnostics In multiple linear regression, we need the linearity assumption to hold for all of the predicting variables, for the model to be a good fit. "For example, if the linearity does not hold with one or more predicting variables, then we could transform the predicting variables to improve the linearity assumption."
In multiple linear regression, a VIF value of 6 for a predictor means that 90% of the variation in that predictor can be modeled by the other predictors.
false See Lesson 3.13: Model Evaluation and Multicollinearity A VIF value of 6 for a predictor means that 83.3% of the variation in that predictor can be modeled by the other predictors in the model.
It is good practice to create a multiple linear regression model using a linearly dependent set of predictor variables.
false See Lesson 3.13: Model Evaluation and Multicollinearity It is good practice to create a multiple linear regression model using a linearly independent set of predicting variables. "XTX is not invertible if the columns of X are linearly dependent, i.e. one predicting variable, corresponding to one column, is a linear combination of the others."
Multicollinearity in multiple linear regression means that the rows in the design matrix are (nearly) linearly dependent.
false See Lesson 3.13: Model Evaluation and Multicollinearity Multicollinearity in multiple linear regression means that the columns in the design matrix are (nearly) linearly dependent.
Multicollinearity among the predicting variables will not impact the standard errors of the estimated regression coefficients.
false See Lesson 3.13: Multicollinearity Multicollinearity in the predicting variables can impact the standard errors of the estimated coefficients. "However, the bigger problem is that the standard errors will be artificially large."
In multiple linear regression, the prediction of the response variable and the estimation of the mean response have the same interpretation.
false See Lesson 3.2.9: Regression Line and Predicting a New Response. In multiple linear regression, the prediction of the response variable and the estimation of the mean response do not have the same interpretation.
A multiple linear regression model contains 6 quantitative predicting variables and an intercept. The number of parameters to estimate in this model is 7.
false See Lesson 3.2: Basic Concepts The number of parameters to estimate in a multiple linear regression model containing 6 quantitative predicting variables and an intercept is 8: 7 regression coefficients (β0,β1,...,β6) and the variance of the error terms (σ2).
What are three problems that variable selection tries to minimize?
high dimensionality, multicollinearity, prediction vs explanatory
The estimated variance of the error terms of a multiple linear regression model with intercept can be obtained by summing up the squared residuals and dividing the sum by n - p , where n is the sample size and p is the number of predictors.
false See Lesson 3.3: Regression Parameter Estimation The estimated variance of the error terms of a multiple linear regression model with intercept should be obtained by summing up the squared residuals and dividing that by n-p-1, where n is the sample size and p is the number of predictors as we lose p+1 degrees of freedom when we estimate the p coefficients and 1 intercept.
The causation of a predicting variable to the response variable can be captured using multiple linear regression on observational data, conditional of other predicting variables in the model.
false See Lesson 3.4 Model Interpretation "This is particularly prevalent in a context of making causal statements when the setup of the regression does not allow so. Causality statements can only be made in a controlled environment such as randomized trials or experiments. "
Conducting t-tests on each β parameter in a multiple linear regression model is the preferable to an F-test when testing the overall significance of the model.
false See Lesson 3.7: Testing for Subsets of Coefficients "We cannot and should not select the combination of predicting variables that most explains the variability in the response based on the t-tests for statistical significance because the statistical significance depends on what other variables are in the model."
In a multiple linear regression model, the adjusted R^2 measures the goodness of fit of the model
false The adjusted R^2 is not a measure of Goodness of fit. R^2 and adjusted R^2 measures the ability of the model and the predictor variable to explain the variation in response variable. Goodness of Fit refers to having all model assumptions satisfied.
In ANOVA, when testing for equal means across groups, the alternative hypothesis is that the means are not equal between two groups for all pairs of means/groups.
false The alternative is that at least one pair of groups have unequal means
Under Poisson regression, the sampling distribution used for a coefficient estimator is a chi-squared distribution when the sample size is large.
false The coefficient estimator follows an approximate normal distribution.
In regularized regression, the penalization is generally applied to all regression coefficients (β0, ... ,βp), where p = number of predictors.
false The shrinkage penalty is applied to β1, . . . , βp, but not to the intercept β0.
The number of parameters that need to be estimated in a logistic regression model with 5 predicting variables and an intercept is the same as the number of parameters that need to be estimated in a standard linear regression model with an intercept and same predicting variables.
false There are no error terms in logistic regression, so we only have parameters for the 6 coefficients in the model. With linear regression, we have parameters for the 6 coefficients in the model as well as the variance of the error terms.
Variable selection is a simple and completely solved statistical problem since we can implement it using the R statistical software.
false Variable selection for a large number of predicting variables is an "unsolved" problem, and variable selection approaches should be tailored to the problem at hand.
It is good practice to perform a goodness-of-fit test on logistic regression models without replications.
false We can only define residuals for binary data with replications, and residuals are needed for a goodness-of-fit test.
When testing a subset of coefficients, deviance follows a chi-square distribution with q degrees of freedom, where q is the number of regression coefficients in the reduced model.
false q is difference between the number of regression coefficients in the full model and the reduced model.
When the number of predicting variables is large, both backward and forward stepwise regressions will always select the same set of variables.
false Backward and forward stepwise regressions will not always select the same set of variables.
It is good practice to perform variable selection based on the statistical significance of the regression coefficients.
false It is not good practice to perform variable selection based on the statistical significance of the regression coefficients.
A logistic regression model has the same four model assumptions as a multiple linear regression model.
false The assumptions of a logistic regression model are: 1. Linearity Assumption: There is a linear relationship between the link function and the predictors 2. Independence assumption: The response variables are independent random variables 3. The link function is the logit function
What kind of variable is a predicting variable and why?
fixed, because it does not change with the response but it is fixed before the response is measured.
Where does uncertainty from estimation come from?
from estimation alone
Where does uncertainty from prediction come from?
from the estimation of regression parameters and from the newness of the observation itself
Cross-validation
leave out some of the data when fitting a model, that is, split the data into two parts. One part, also called a training data, will be used to fit the model given a specific lambda, and thus give the estimated regression coefficients given that lambda constant.
what is the coefficient interpretation of a GLM (poisson)?
log ratio of the rate with an increase with one unit in the predicting variable.
A study was conducted to measure the effect of a fungicide treatment on the survival rate of botrytis blight. Botrytis blight samples were divided into 20 groups, each consisting of about 100 samples and exposed to different levels of chemicals in a fungicide. The output of a logistic regression model is below, where concS represents the concentration of a sulfur in the fungicide and concCu represents the concentration of a copper in the fungicide. Use it to answer the following multiple-choice questions. Call: glm(formula = cbind(Survived, Died) ~ concS + concCu,family = "binomial", data = data) Deviance Residuals: Min 1Q Median 3Q Max -9.5366 -2.4594 0.1223 3.9710 6.3566 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) 3.58770 0.22958 15.63 <2e-16 *** concS -4.32735 0.26518 16.32 <2e-16 *** concCu -0.27483 0.01784 15.40 <2e-16 *** Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 718.76 on 19 degrees of freedom Residual deviance: 299.43 on 17 degrees of freedom AIC: 363.53 The p-value for a goodness-of-fit test using the deviance residuals for the regression can be obtained from which of the following? pchisq(419.33,2, lower.tail =FALSE) pchisq(363.53,3, lower.tail =FALSE) pchisq(299.43,17, lower.tail =FALSE) pchisq(718.76,19, lower.tail =FALSE)
pchisq(299.43,17, lower.tail =FALSE) The goodness of fit test uses the residual deviance (299.43) and corresponding degrees of freedom (17) as the test statistic for the chi-squared test.
For goodness of fit test, we compare the likelihoods of the...?
saturated model versus the fitted model
Ridge regression does not perform variable selection. It only...
shrinks the coefficients to zero but it does not FORCE coefficients to be zero as needed in variable selection.
training error
simply use the data to fit the model then compute the classifier from each response in the data and take the proportion of the responses we misclassified
Forward stepwise tends to select...
smaller models
HW3 Multi Regression
spacing
L1 penalty measures...
sparsity
The test statistic for the goodness of fit test is?
sum of squared deviances
If we replace the unknown variance with its estimator, sigma^2=MSE, for PREDICTION, the sampling distribution becomes...
t distribution with n-p-1 DF
The alternative hypothesis of ANOVA can be stated as, the means of all pairs of groups are different the means of all groups are equal the means of at least one pair of groups is different None of the above
the means of at least one pair of groups is different See 2.4 Test for Equal Means "Using the hypothesis testing procedure for equal means, we test: The null hypothesis, which that the means are all equal (mu 1 = mu 2...=mu k) versus the alternative hypothesis, that some means are different. Not all means have to be different for the alternative hypothesis to be true -- at least one pair of the means needs to be different."
Goodness of fit tests the null hypothesis that
the model fits the data
Deviance
the test statistic is the difference of the log likelihood under the reduced model and the log likelihood under the full model for testing the subset of coefficients
If we have a positive value for B1,....
then that's consistent with a direct relationship between the predicting variable x and the response variable y.
Assuming the model is a good fit, the residuals in simple linear regression have constant variance.
true
Elastic net regression uses both penalties of ridge and lasso regression and hence combines the benefits of both.
true
If a Poisson regression model does not have a good fit, the relationship between the log of the expected rate and the predicting variables might be not linear.
true
If a predicting variable is categorical with 5 categories in a linear regression model without intercept, we will include 5 dummy variables in the model.
true
In a simple linear regression model, given a significance level α , if the ( 1 − α ) % confidence interval for a regression coefficient does not include zero, we conclude that the coefficient is statistically significant at the α level.
true In a simple linear regression model, given a significance level α , if the ( 1 − α ) % confidence interval for a regression coefficient does not include zero, we conclude that the coefficient is statistically significant at the α level.
It is required to standardize or rescale the predicting variables when performing regularized regression.
true Regularized regression requires standardization or scaling of the predicting variables.
A negative value of β 1 is consistent with an inverse relationship between the predictor variable and the response variable.
true See 1.2 Estimation Method "A negative value of β 1 is consistent with an inverse relationship"
The pooled variance estimator, s p o o l e d 2, in ANOVA is synonymous with the variance estimator, σ ^ 2, in simple linear regression because they both use mean squared error (MSE) for their calculations.
true See 1.2 Estimation Method for simple linear regression See 2.2 Estimation Method for ANOVA The pooled variance estimator is, in fact, the variance estimator.
Under the normality assumption, the estimator for β 1 is a linear combination of normally distributed random variables.
true See 1.4 Statistical Inference "Under the normality assumption, β 1 is thus a linear combination of normally distributed random variables... β ^ 0 is also linear combination of random variables"
The prediction interval of one member of the population will always be larger than the confidence interval of the mean response for all members of the population when using the same predicting values.
true See 1.7 Regression Line: Estimation & Prediction Examples "Just to wrap up the comparison, the confidence intervals under estimation are narrower than the prediction intervals becausethe prediction intervals have additional variance from the variation of a new measurement."
If the model assumptions hold, then the estimator for the variance, σ ^ 2, is a random variable.
true See 1.8 Statistical Inference We assume that the error terms are independent random variables. Therefore, the residuals are independent random variables. Since σ ^ 2 is a combination of the residuals, it is also a random variable.
An ANOVA model with a single qualitative predicting variable containing k groups will have k + 1 parameters to estimate.
true See 2.2 Estimation Method We have to estimate the means of the k groups and the pooled variance estimator, s p o o l e d 2.
The mean sum of squared errors in ANOVA measures variability within groups.
true See 2.4 Test for Equal Means MSE = within-group variability
If the constant variance assumption in ANOVA does not hold, the inference on the equality of the means will not be reliable.
true See 2.8 Data Example "This is important since without a good fit, we cannot rely on the statistical inference." Only when the model is a good fit, i.e. all model assumptions hold, can we rely on the statistical inference.
If the pairwise comparison interval between groups in an ANOVA model includes zero, we conclude that the two means are plausibly equal.
true See 2.8 Data Example If the comparison interval includes zero, then the two means are not statistically significantly different, and are thus, plausibly equal.
Cook's distance (Di) measures how much the fitted values in a multiple linear regression model change when the ith observation is removed.
true See Lesson 3.11: Assumptions and Diagnostics "This is the distance between the fitted values of the model with all the observations versus the fitted values of the model discarding the i-th observation from the data used to fit the model. "
The presence of certain types of outliers, such as influential points, can impact the statistical significance of some of the regression coefficients.
true See Lesson 3.11: Assumptions and diagnostics Outliers that are influential can impact the statistical significance of the beta parameters.
An example of a multiple linear regression model is Analysis of Variance (ANOVA).
true See Lesson 3.2 Basic Concepts "Earlier, we contrasted the simple linear regression model with the ANOVA model... Multiple linear regression is a generalization of both models."
If the residuals are not normally distributed, we can model the transformed response variable instead, where a common transformation for normality is the Box-Cox transformation.
true See Lesson 3.3.11: Assumptions and Diagnostics If the normality assumption does not hold, we can use a transformation that normalizes the response variable such as Box-Cox transformation.
linear regression model has high explanatory power if the coefficient of determination is close to 1.
true See Lesson 3.3.13: Model Evaluation and Multicollinearity If R2 is close to 1, almost all of the variability in Y can be explained by the linear regression model; hence, the model has high explanatory power.
In the case of multiple linear regression, controlling variables are used to control for sample bias.
true See Lesson 3.4: Model Interpretation "Controlling variables can be used to control for bias selection in a sample."
For a given predicting variable, the corresponding estimated regression coefficient will likely be different in a conditional model versus a marginal model.
true See Lesson 3.4: Model Interpretation "Importantly, the estimated regression coefficients for the conditional and marginal relationships can be different, not only in magnitude but also in sign or direction of the relationship."
In multiple linear regression, the estimated regression coefficient corresponding to a quantitative predicting variable is interpreted as the estimated expected change in the response variable when there is a change of one unit in the corresponding predicting variable holding all other predictors fixed.
true See Lesson 3.4: Model Interpretation "The estimated value for one of the regression coefficient βi represents the estimated expected change in y associated with one unit of change in the corresponding predicting variable, Xi, holding all else in the model fixed."
A partial F-Test can be used to test whether the regression coefficients associated with a subset of the predicting variables in a multiple linear regression model are all equal to zero.
true See Lesson 3.7: Testing for Subsets of Regression Parameters We use the Partial F-test to test the null hypothesis that the regression coefficients associated to a subset of the predicting variables are all equal to zero. The alternative hypothesis is that at least one of these regression coefficients is not zero.
Simpson's Paradox occurs when a coefficient reverses its sign when used in a marginal versus a conditional model.
true Simpson's paradox: Reversal of an association when looking at a marginal relationship versus a conditional relationship.
Generalized linear models, like logistic regression, use a Wald test to determine the statistical significance of the coefficients.
true The coefficient estimates follow an approximate normal distribution and a z-test, also known as a Wald test, is used to determine their statistical significance.
For logistic regression, if the p-value of the deviance test for goodness-of-fit is large, then it suggests that the model is a good fit.
true The null hypothesis is that the model fits the data. So large p-values suggests that the model is a good fit.
With Poisson regression, the variance of the response is not constant.
true V(Y|x_1,...x_p)=exp(beta_0 + beta_1 x_1 + ... + beta_p x_p)
Although there are no error terms in a logistic regression model using binary data with replications, we can still perform residual analysis.
true We can perform residual analysis on the Pearson residuals or the Deviance residuals.
The training risk is not an unbiased estimator of the prediction risk.
true The training risk is a biased estimator of the prediction risk.
When selecting variables for explanatory purpose, one might consider including predicting variables which are correlated if it would help answer your research hypothesis.
true When the objective is to explain the relationship to the response, one might consider including the predicting variables even they are correlated.
What does it mean if 0 is NOT included in the CI?
we conclude that Bj IS statistically significant
What does it mean if 0 is included in the CI?
we conclude that Bj is NOT statistically significant
pairwise comparison
we estimate the difference in the means (for example, a pair: meani and meanj) as a difference between their corresponding means
If we add the bias squared and the variance,
we get Mean Squared Error (MSE)
What does a cluster of residuals mean on a residual plot?
we have correlated errors
When B1 is close to zero...
we interpret that there is not a significant association between predicting variables, between the predicting variable x, and the response variable y.
If the t-value is large...
we reject the null hypothesis and conclude that the coefficient is statistically significant.
Why do we lose 1 DF for s^2?
we replace mu with zbar
Why do we lose 2 DF for sigma^2?
we replaced two parameters, B0 and B1
Backward stepwise regression
we start with all predictors, the full model and drop one predictor at a time.
Forward stepwise regression
we start with no predictor or with a minimum model, and add one predictor at a time
When do we reject the null hypothesis for the overall regression test in regards to the p value?
when the P-value is small, indicating that the overall regression has explanatory power.
A model selection method that can be performed in the R statistical software given a set of controlling factors that form the minimum or starting model.
Stepwise regression
Forward stepwise tends to result in smaller models that backward stepwise regression.
True
Given a model of p predicting variables, there are p^2 models to choose from.
True
If there is a group of correlated variables, lasso tends to select only one of the variables in the group.
True
If you had a model with multicollinearity that you wanted to preserve, you'd use ridge regression over lasso.
True
In cases where p>n, lasso selects only up to n variables.
True
In regularized regression, different lambdas result in different models.
True
In regularized regression, lambda is a constant.
True
The L1 penalty results in ridge regression.
False
The penalty for lasso regression is not a sparsity penalty.
False
The selected variables using variable selection approaches are the only variables that explain the response variable.
False
We need not be wary of over interpretation in MLR.
False
Variable selection methods is performed by
Balancing the bias-variance trade-off.
Why is training risk a biased estimate of prediction risk?
Because we use the data twice.
What does training risk correct for?
Bias
In GLMs the main reason one does not use LSE to estimate model parameters is the potential constrained in the parameters.
False - The potential constraint in the parameters of GLMs is handled by the link function.
The backward elimination requires a pre-set probability of type II error
False - Type I error
Can be used to explain variability in the response variable.
Explanatory variable
All regularized regression approaches will select the same model.
False
Forward stepwise is more computationally expensive than backward.
False
If a predicting variables is selected to be in the model, we conclude that the predicting variable has a causal relationship with the response variable.
False
In regularized regression, the bigger the lambda the smaller the complexity penality.
False
It is always feasible to apply a model search for all possible combinations of the predicting variables.
False
LOOCV penalizes complexity more than Mallow's CP
False
Lasso regression has a closed from expression for the estimated coefficients.
False
Mallows Cp works even when p>n.
False
Once lasso has identified the predicting variables, it is not necessary to use OLS to obtain the regression coefficients.
False
Ridge regression is used for variable selection.
False
The AIC and BIC cannot be used in selecting variables for generalized linear models.
False
It is possible to select a model to include variables that are not statistically significant, even though that model will provide the best prediction.
True
Models with many predictors have what?
Low bias, high variance.
L0 penalty is equivalent to searching through all the models.
True
The minus expected log likelihood function is also known as what?
Prediction risk
Can be used to predict variability in the response regardless of their predictive power.
Predictive variable
Mallows CP uses estimated variance based on what?
The full model
Forward stepwise regression adds one variable to the model at a time starting with a minimum model.
True
Alpha balances between ridge and lasso in elasticnet.
True
Complexity is equivalent to a large model with many predicting variables.
True
Cross validation is used to find the best lambda for regularized regression problems.
True
For BIC, we need to replace sigma^2 with an estimate from the submodel S.
True
For logistic regression and Poisson regression, the training risk is the sum of square deviances for the submodel S.
True
We cannot obtain prediction risk because of what?
We don't have future observations at the time of prediction.
