Chapter 9

What is GBM Mechanism?

01:GBM predicts the residuals or errors of prior models and then sums them to make the final prediction 02:One weak learner is added at a time and existing weak learners in the model are left unchanged 03:GBM repetitively leverages the pattern in residuals and strenghthens a model with weak predictions. 04:Model is stopped when residuals do not have any pattern that can be modelled

What are XGBoost parameters?

1. General Parameters: number of threads 2. Booster Parameters: Step size; regulation 3. Task Parameters: Objective;Evaluation metric

What are the techniques to assess model performance?

1. Train/Test split 2. Cross Validation split

What are factors that depend on performance of Ensemble learning?

1.Robustness and 2.Accuracy. Robustness- Ensemble models incorporate the predictions from all the base learners. Accuracy - Ensemble models deliver accurate predictions and have improve performance.

What is Adaboost Algorithm?

A weal classifier is prepared on the training data using the weighted samples. Only binary classification problems are supported Every decision stump makes one decision on one input variable and outputs a +1.0 or -1.0 value for the first or second class value. Misclassification rate (error) = (Correct - N)/N

What is Bagging?

Bagging or Bootstrap aggregation reduces variance of an estimate by taking mean of multiple estimates. There are 3 Steps to perform bagging: 1: Create Randomly sampled data sets of the original training data(bootstrapping) 2:Build and fit several classifiers to each of these diverse copies 3:Take the average of all the predictions to make a final overall prediction.

What is Boosting?

Boosting is a sequential process where errors of previous models is corrected by each subsequent model. Example : Different types of Body pain X : Head ace Y : Leg pain Z: Stomach pain. Depending of results of weighted opinions doctor will suggest diagnosis sequentially. Boosting reducing bias by training weak learners sequentially, each trying to correct its predecessors.

What is AdaBoost and how it works?

Boosting is a technique of changing weak learners into strong learners.Each new tree is a fit on a modified version of the original dataset. Adaboost is the first boosting algorithm to be adapted in solving practices. It helps in mixing multiple weak classifiers into one strong classifier.

What is Ideology of Ensemble Learning

Certain models do well in modelling in one aspects of the data, while others do well in modelling another. Instead of Learning a large complex model learn several simple models and combine their output to produce the final decision. The combined strength of the model offsets individual model variances and biases. It provides a composite prediction where the final accuracy is better than accuracy of individual models

What are differences between Train/Test Split VS Cross-Validation?

Cross Validation: More accurate estimate of out of sample accuracy More efficient use of data (every observation is used for both training and testing) Train/Test Split: Run K-times faster than K-fold cross validation. Simpler to examine the detailed results of the testing process.

What is Train/Test Split ?

Divide the training dataset Training on first training dataset Test on the second set

What is Ensemble Learning?

Ensemble Techniques combine individual models together to improve the stability and predictive power of the model. -This Technique permits higher predictive -performance -It combines multiple machine learning models into one predictive model Decrease Variance using Bagging Decrease Bias using Boosting Improve prediction using Staking

What are different types of Ensemble methods?

Ensemble methods can be classified in to two groups 1. Sequential Ensemble methods 2. Parallel Ensemble methods

What is Gradient Boosting?

Gradient Boosting involves 3 elements : 1. A loss function to be optimized 2.A weal learner to make predictions 3.An additive model to add weak learners to minimize the loss function. GBM minimizes the loss function(MSE) of a model by adding a weak learners using a gradient descent procedure.

What is Sequential Ensemble methods?

In sequential ensemble methods base learners are generated consecutively that is adaboost. Basic motivation is to use the dependence between base learners. By weighting previously mislabeled examples with higher weight the overall performance can be boosted.

How to create an Ensemble learning model?

It can be created by combining all weak learners to form an ensemble or well chosen strong and diverse models

What is Model Averaging Ensemble?

Model averaging is the approach to ensemble learning where each ensemble learner member contributes an equal amount to the final prediction Like Decision Tree + Logistic Regression + SVM Regression - Ensemble prediction - Average of Member prediction - p1+p2+p3/3 A Class label is predicted using mode of member predictions The class probability is calculated as the argmax of the summed probabilities for each class label. Equal weights are assigned to different models despite some models performing better than others

How Random forest is an Ensemble Learning model?

Random forest is good example of ensemble learning model. Random forest technique combine various decision trees to produce a more generalized model. By reducing notorious of over fitting of decision trees. Random forest are utilized to produced de-correlated decision trees. Random forest create random subset of the features. Smaller trees are built using these subsets creating tree diversity. To overcome over fitting, diverse set of decision trees are required.

What is cross validation split?

Sets of Train/Test splits are created. Accuracy of each split is checked. Results are averaged

What is adaboost flowchart?

Step1: Adaboost selects a training dataset randomly step2: It iteratively trains the Adaboost ML model Step3:It assigns a higher weight to wrongly classified observations. Step4:It assigns weight to the trained classifier in each iteration according to the accuracy of the classifier. Step5: this process is iterates until the complete training data fits without any error

Explain steps in GBM Algorithm?

Step1: Fit a simple regression or classification model Step2: Calculate error residuals(Actual Value - Predicted Value) Step3: Fit a new model on error residuals as target variable with same input variables Step4: Add the predicted residuals to the previous predictions Step5: Fit another model on residuals that are remaining and repeat steps 2 and 5 until model is overfit or sum of the residuals become constant

What are the Steps of Adaboost algorithm?

Step1: Initially each data point is weighted equally with weight. Wi = 1/n where n is the number of samples. Step2: A classifier H1 is picked up that classifies the data with minimal error rate. Step3: The weighting factor is dependent on errors(et) caused by the H1 classifier. Step4: weight after time t is given

How Boosting works?

Step1: Train a classifier H1 that best classifies the data with respect to accuracy Step2:Identify the regions where H1 produces errors, add weights to them, and produce a H2 classifier Step3: Aggregate those samples for which H1 produces a different result from H2 and produce a H3 classifier. Repeat step2 for new classifier

What is Parallel Ensemble methods?

are applied wherever the base learners are generated in parallel for ex: random forest since the errors are often reduced dramatically by averaging the basic motivation is to use independence between base learners

What is XGBoost?

eXtrement Gradient Boosting is a library for developing fast and high-performance gradient boosting tree models. Custom tree building algorithm Interfaces for python and R and can be executed on Yarn. Used for Classification,Regression,Ranking with custom loss functions. XGBoost is extensively used in ML competitions as it is almost 10 times faster than other gradient boosting techniques.

What is Weighted Average Ensemble?

is an extension of a model averaging on ensemble where the contribution of each member to the final prediction is weighted by performance of the model. the model weights are small positive values and the sum of all weights equal to 1. w1+w2+w3 = 1 p = w1*p1+w2*p2+w3*p3