How to optimise size depth of trees in XGBoost in R?

Recipe Objective

Classification and regression are supervised learning models that can be solved using algorithms like linear regression / logistics regression, decision tree, etc. But these are not competitive in terms of producing a good prediction accuracy.Ensemble techniques, on the other hand, create multiple models and combine them into one to produce effective results.

Bagging, boosting, random forest, are different types of ensemble techniques. Boosting is a sequential ensemble technique in which the model is improved using the information from previously grown weaker models. This process is continued for multiple iterations until a final model is built which will predict a more accurate outcome. ​

List of Classification Algorithms in Machine Learning

There are 3 types of boosting techniques: ​

1. Adaboost
2. Gradient Descent.
3. Xgboost

Recently, researchers and enthusiasts have started using ensemble techniques like XGBoost to win data science competitions and hackathons. It outperforms algorithms such as Random Forest and Gadient Boosting in terms of speed as well as accuracy when performed on structured data. ​

XGBoost uses ensemble model which is based on Decision tree. A simple decision tree is considered to be a weak learner. The algorithm build sequential decision trees were each tree corrects the error occuring in the previous one until a condition is met.

In this recipe, we will discuss how to build and optimise size of the tree in XGBoost.. ​

STEP 1: Importing Necessary Libraries

library(caret) # for general data preparation and model fitting library(tidyverse)

STEP 2: Read a csv file and explore the data

The dataset attached contains the data of 160 different bags associated with ABC industries.

The bags have certain attributes which are described below: ​

1. Height – The height of the bag
2. Width – The width of the bag
3. Length – The length of the bag
4. Weight – The weight the bag can carry
5. Weight1 – Weight the bag can carry after expansion

The company now wants to predict the cost they should set for a new variant of these kinds of bags. ​

data <- read.csv("R_360_Data_1.csv") glimpse(data)

Rows: 159
Columns: 6
$ Cost     242, 290, 340, 363, 430, 450, 500, 390, 450, 500, 475, 500,...
$ Weight   23.2, 24.0, 23.9, 26.3, 26.5, 26.8, 26.8, 27.6, 27.6, 28.5,...
$ Weight1  25.4, 26.3, 26.5, 29.0, 29.0, 29.7, 29.7, 30.0, 30.0, 30.7,...
$ Length   30.0, 31.2, 31.1, 33.5, 34.0, 34.7, 34.5, 35.0, 35.1, 36.2,...
$ Height   11.5200, 12.4800, 12.3778, 12.7300, 12.4440, 13.6024, 14.17...
$ Width    4.0200, 4.3056, 4.6961, 4.4555, 5.1340, 4.9274, 5.2785, 4.6...

summary(data) # returns the statistical summary of the data columns

Cost            Weight         Weight1          Length     
 Min.   :   0.0   Min.   : 7.50   Min.   : 8.40   Min.   : 8.80  
 1st Qu.: 120.0   1st Qu.:19.05   1st Qu.:21.00   1st Qu.:23.15  
 Median : 273.0   Median :25.20   Median :27.30   Median :29.40  
 Mean   : 398.3   Mean   :26.25   Mean   :28.42   Mean   :31.23  
 3rd Qu.: 650.0   3rd Qu.:32.70   3rd Qu.:35.50   3rd Qu.:39.65  
 Max.   :1650.0   Max.   :59.00   Max.   :63.40   Max.   :68.00  
     Height           Width      
 Min.   : 1.728   Min.   :1.048  
 1st Qu.: 5.945   1st Qu.:3.386  
 Median : 7.786   Median :4.248  
 Mean   : 8.971   Mean   :4.417  
 3rd Qu.:12.366   3rd Qu.:5.585  
 Max.   :18.957   Max.   :8.142   

dim(data)

159 6

STEP 3: Train Test Split

# createDataPartition() function from the caret package to split the original dataset into a training and testing set and split data into training (80%) and testing set (20%) parts = createDataPartition(data$Cost, p = .8, list = F) train = data[parts, ] test = data[-parts, ]

STEP 4: Building and Optimising the depth of xgboost tree

We will use caret package to perform Cross Validation and Hyperparameter tuning (nround- Number of trees and max_depth) using grid search technique. Firstly, we will use the trainControl() function to define the method of cross validation to be carried out and then use train() function.

Syntax: train(formula, data = , method = , trControl = , tuneGrid = )

where:

1. formula = y~x1+x2+x3+..., where y is the independent variable and x1,x2,x3 are the dependent variables
2. data = dataframe
3. method = Type of the model to be built
4. trControl = Takes the control parameters. We will use trainControl function out here where we will specify the Cross validation technique.
5. tuneGrid = takes the tuning parameters and applies grid search CV on them

set.seed(50) # specifying the CV technique which will be passed into the train() function later and number parameter is the "k" in K-fold cross validation train_control = trainControl(method = "cv", number = 5) # Customsing the tuning grid gbmGrid <- expand.grid(max_depth = c(3, 5, 7), nrounds = (1:10)*50, # number of trees # default values below eta = 0.3, gamma = 0, subsample = 1, min_child_weight = 1, colsample_bytree = 0.6) # training a XGboost Regression tree model while tuning parameters model = train(Cost~., data = train, method = "xgbTree", trControl = train_control, tuneGrid = gbmGrid) # summarising the results print(model)

eXtreme Gradient Boosting 

129 samples
  5 predictor

No pre-processing
Resampling: Cross-Validated (5 fold) 
Summary of sample sizes: 103, 104, 103, 103, 103 
Resampling results across tuning parameters:

  max_depth  nrounds  RMSE      Rsquared   MAE     
  3           50      61.55397  0.9727063  37.89925
  3          100      61.65331  0.9727531  37.65524
  3          150      61.83832  0.9725518  37.86967
  3          200      61.88306  0.9725015  37.90947
  3          250      61.89125  0.9724915  37.93032
  3          300      61.88905  0.9724920  37.92371
  3          350      61.88783  0.9724928  37.92007
  3          400      61.88708  0.9724929  37.91850
  3          450      61.88686  0.9724930  37.91768
  3          500      61.88674  0.9724930  37.91731
  5           50      61.68127  0.9722249  37.83729
  5          100      61.88318  0.9720744  37.95097
  5          150      61.89724  0.9720586  37.96368
  5          200      61.89829  0.9720578  37.96544
  5          250      61.89838  0.9720577  37.96551
  5          300      61.89838  0.9720577  37.96551
  5          350      61.89838  0.9720577  37.96551
  5          400      61.89838  0.9720577  37.96551
  5          450      61.89838  0.9720577  37.96551
  5          500      61.89838  0.9720577  37.96551
  7           50      62.74921  0.9731129  38.70850
  7          100      62.76334  0.9731210  38.74388
  7          150      62.76427  0.9731209  38.74426
  7          200      62.76428  0.9731209  38.74427
  7          250      62.76428  0.9731209  38.74427
  7          300      62.76428  0.9731209  38.74427
  7          350      62.76428  0.9731209  38.74427
  7          400      62.76428  0.9731209  38.74427
  7          450      62.76428  0.9731209  38.74427
  7          500      62.76428  0.9731209  38.74427

Tuning parameter 'eta' was held constant at a value of 0.3
Tuning

Tuning parameter 'min_child_weight' was held constant at a value of 1

Tuning parameter 'subsample' was held constant at a value of 1
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were nrounds = 50, max_depth = 3, eta
 = 0.3, gamma = 0, colsample_bytree = 0.6, min_child_weight = 1 and subsample
 = 1.

Note: RMSE was used select the optimal model using the smallest value. And the final model consists of 50 trees and depth of 3.

STEP 5: Make predictions on the final xgboost model

We use our final xgboost model to make predictions on the testing data (unseen data) and predict the 'Cost' value and generate performance measures.

#use model to make predictions on test data pred_y = predict(model, test) # performance metrics on the test data test_y = test[, 1] mean((test_y - pred_y)^2) #mse - Mean Squared Error caret::RMSE(test_y, pred_y) #rmse - Root Mean Squared Error

4856.9162586605
69.6915795391416