Chapter 8 Trees

Greedy or Not Greedy? Boosting

Boosting is greedy because it focuses intensively on reducing error in a sequential manner,

(T/F) A decision tree with 𝑛 leaves has 𝑛 branches

F

(T/F) In bagging, the trees are grown independently on random samples of the observations, and thus the trees tend to be quite dissimilar to each other.

F. They tend to be quite similar

(T/F) Random forests tend to outperform bagging when many of the predictors are irrelevant to explaining the response variable.

F. We do random forest because one predictor dominates others. In this case, random forests will be forced to use shit predictors

(T/F) In practice, estimating the cross-validation error for every possible subtree would not be cumbersome, despite the vast number of possible subtrees.

F. there is an extremely large number of possible subtrees.

(T/F) A stump is a decision tree with no leaves.

False, it will have two terminal nodes/leaves

(T/F) Boosted model: When 𝑑=1, the boosted model resembles an multiplicative model.

False. d=1 is known as an additive model. 𝑑 controls the complexity of the boosted model. It is also known as the interaction depth.

(T/F) Pruning random forests reduces overfitting

False. variance reduction is done by averaging the results of multiple trees

(T/F) In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value. Does the split could increase Gini Index or entropy?

No. It decreases the Gini/ Entropy since those impurity measures are more sensitive to node purity

(T/F) A better strategy is to grow a very large tree, and then prune it back to obtain a subtree with fewer splits that might lead to lower variance and better interpretation at the cost of a little bias.

T

(T/F) The entropy and Gini index are similar measures numerically.

True

Greedy or Not Greedy? Bagging

Bagging is not greedy as it builds independent models in parallel without focusing sequentially on correcting previous errors.

(T/F) In boosting, we only use the bootstrapped data, and do not draw any random samples. The trees are grown successively, using a slow learning approach.

F. Boosting uses ORIGINAL DATA, and does not draw any random samples

(T/F) Bagging requires bootstrapping, but random forests do not.

F. Both do

(T/F) For decision trees, categorical variables should be handled using dummy variables.

F. Dummy variables are not necessary since every possible grouping of the classes into two groups is considered when splitting.

(T/F) For a boosting model, create as many trees as possible.

F. In boosting, 𝑏 is a flexibility measure, so overfitting can occur if it is too large. Use Cross validation to choose b.

(T/F) Each split on each tree in random forests is performed using a fixed subset of the features, thereby decorrelating the trees, and leading to a more thorough exploration of model space relative to bagging.

F. It uses a random subset of fatures

(T/F) The larger the tuning parameter, the larger the selected subtree tends to be.

F. Large penalty=small tree

(T/F) A large value of the Gini index indicates that a node contains predominantly observations from a single class.

F. Look at vertical axis. A small value of the Gini index/Entropy indicates a node is mostly from a single class

(T/F) Trees are not an attractive choice of weak learner for an ensemble method.

F. Pros: their flexibility and ability to handle predictors of mixed types

(T/F) Pruning a tree decreases its bias.

F. Pruning a tree increases its bias and decreases variance.

(T/F) Bagging tends to outperform random forests when one of the predictors is much stronger than other predictors

F. Random forests will win in this case

(T/F) Cost Complexity: As the tuning parameter 𝛼 value increases, the error sum of squares plus the number of terminal nodes will be minimized for a smaller subtree.

F. The RSS plus the number of terminal nodes times the tuning parameter 𝜆 |T| will tend to be minimized for a smaller subtree.

(T/F) When using cost complexity pruning, the penalty decreases as the number of terminal nodes in the subtree increases.

F. The penalty is 𝜆| 𝑇 |. Both increases

(T/F) In the classification setting, residual sum of squares is used as a criterion for making the binary splits.

F. The quality of a split is evaluated based on node purity(Gini and entropy)

(T/F) The way that bagging reduces variance is by pruning the bagged trees

F. They do it by averaging the bagged trees

Greedy or Not Greedy? Backward stepwise selection

Greedy: choosing the best variable to remove at each step.

Greedy or Not Greedy? Forward stepwise selection

Greedy: choosing the best variable to remove at each step.

Greedy or Not Greedy? Recursive binary splitting

Greedy: selects the best variable to split on at each step of the tree-building process

A (small/large) large value of d, tree depth, in a boosted model means a more complex model

Large value of d means more complex

A large d(tree depth) may require a (LARGE/SMALL) value of B(number of trees) to achieve good performance.

SMALL value. if d is small, we need more B because the trees are less complex.

(T/F) A seemingly worthless split early on in the tree might be followed by a very good split, that leads to a large reduction in residual sum of squares later on.

T

(T/F) Bagging, random forests, boosting, and Bayesian additive regression trees are ensemble methods for which the simple building block is a regression or a classification tree.

T

(T/F) Bootstrap aggregation, or bagging, is a general-purpose procedure for reducing the variance of a statistical learning method.

T

(T/F) For a classification tree, we predict that each observation belongs to the most commonly occurring class of training observations in the region to which it belongs.

T

(T/F) Instead of exhaustively considering every potential subtree, weakest link pruning involves evaluating a sequence of trees indexed by a nonnegative tuning parameter 𝛼.

T

(T/F) Pruning: Given a subtree, we can estimate its test error using cross-validation or the validation set a

T

(T/F) The goal of tree pruning is to select a subtree that leads to the lowest test error rate.

T

(T/F) An ensemble method is an approach that combines many simple building ensemble block models in order to obtain a single and potentially very powerful model.

T. By aggregating the predictions of individual models, an ensemble model can exploit the strengths of each base model while mitigating their weaknesses

(T/F) Single decision trees models generally have higher variance than random forest models

T. Predicted responses will vary a lot since its not averaged out.

(T/F) The number of branches is not less than the number of internal nodes.

T. Single node tree(the average of all observation)

(T/F) Weakest link pruning/cost complexity is a method used to select a limited number of subtrees from a larger set of possibilities

T. This approach focuses on identifying and pruning the weakest links or least important branches in the decision tree, resulting in a simpler and more interpretable model.

(T/F) These simple building block models are sometimes known as weak learners, since they may lead to mediocre predictions on their own.

T. their collective predictions can result in a strong learner.

(T/F) For a boosting model, choose a small shrinkage parameter to enable slow learning.

T. 𝜆 is a small positive number that controls the amount that each tree contributes to the overall mode. a value close to 0 indicates slow learning

(T/F) Pruning a tree increases its residual sum of squares.

True. More nodes=more accurate. Less nodes=less accurate

(T/F) OOB estimation can be used for random forest to estimate test error

True. Random forests are based on bootstrapped samples.

(T/F) In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value. Does the split could increase confidence in the predicted value?

Yes. The increased node purity that results from this split increases our confidence in the predicted value,