How to optimise learning rates in XGBoost example 2 in python

This recipe helps you optimise learning rates in XGBoost example 2 in python

Recipe Objective

Many a times while working on a dataset and using a Machine Learning model we don"t know which set of hyperparameters will give us the best result. Passing all sets of hyperparameters manually through the model and checking the result might be a hectic work and may not be possible to do.

To get the best set of hyperparameters we can use Grid Search. Grid Search passes all combinations of hyperparameters one by one into the model and check the result. Finally it gives us the set of hyperparemeters which gives the best result after passing in the model.

So this recipe is a short example of how can optimise multiple parameters in XGBoost.

Sentiment Analysis Project on eCommerce Product Reviews with Source Code

Step 1 - Import the library - GridSearchCv

from sklearn import datasets from sklearn.model_selection import train_test_split from xgboost import XGBClassifier from sklearn.model_selection import GridSearchCV from sklearn.model_selection import StratifiedKFold import numpy

Here we have imported various modules like numpy, test_train_split, datasets, XGBClassifier and GridSearchCV from differnt libraries. We will understand the use of these later while using it in the in the code snipet.
For now just have a look on these imports.

Step 2 - Setup the Data

Here we have used datasets to load the inbuilt wine dataset and we have created objects X and y to store the data and the target value respectively. dataset = datasets.load_wine() X = dataset.data y = dataset.target X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)

Step 3 - Using Model

Here, we are using XGBClassifier as a Machine Learning model to use GridSearchCV. So we have created an object model. model = XGBClassifier()

Step 5 - Parameters to be optimized

In XGBClassifier we want to optimise learning rate by GridSearchCV. So we have set the parameter as a list of values form which GridSearchCV will select the best value of parameter. n_estimators = [100, 200, 300, 400, 500] learning_rate = [0.0001, 0.001, 0.01, 0.1] param_grid = dict(learning_rate=learning_rate, n_estimators=n_estimators) kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)

Step 6 - Using GridSearchCV and Printing Results

Before using GridSearchCV, lets have a look on the important parameters.

  • estimator: In this we have to pass the models or functions on which we want to use GridSearchCV
  • param_grid: Dictionary or list of parameters of models or function in which GridSearchCV have to select the best.
  • Scoring: It is used as a evaluating metric for the model performance to decide the best hyperparameters, if not especified then it uses estimator score.
  • cv: It signifies the number of splits in cross validation.

Making an object grid_search for GridSearchCV and fitting the dataset i.e X and y grid_search = GridSearchCV(model, param_grid, scoring="neg_log_loss", n_jobs=-1, cv=kfold) grid_result = grid_search.fit(X, y) Now we are using print statements to print the results. It will give the values of hyperparameter as a result. print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_)) means = grid_result.cv_results_["mean_test_score"] stds = grid_result.cv_results_["std_test_score"] params = grid_result.cv_results_["params"] for mean, stdev, param in zip(means, stds, params): print("%f (%f) with: %r" % (mean, stdev, param)) As an output we get:

Best: -0.077744 using {"learning_rate": 0.1, "n_estimators": 200}
-1.086577 (0.000539) with: {"learning_rate": 0.0001, "n_estimators": 100}
-1.074744 (0.001073) with: {"learning_rate": 0.0001, "n_estimators": 200}
-1.063105 (0.001605) with: {"learning_rate": 0.0001, "n_estimators": 300}
-1.051657 (0.002128) with: {"learning_rate": 0.0001, "n_estimators": 400}
-1.040397 (0.002643) with: {"learning_rate": 0.0001, "n_estimators": 500}
-0.986714 (0.005129) with: {"learning_rate": 0.001, "n_estimators": 100}
-0.891275 (0.009533) with: {"learning_rate": 0.001, "n_estimators": 200}
-0.808661 (0.013511) with: {"learning_rate": 0.001, "n_estimators": 300}
-0.736632 (0.016334) with: {"learning_rate": 0.001, "n_estimators": 400}
-0.673480 (0.018467) with: {"learning_rate": 0.001, "n_estimators": 500}
-0.443098 (0.032668) with: {"learning_rate": 0.01, "n_estimators": 100}
-0.236960 (0.048792) with: {"learning_rate": 0.01, "n_estimators": 200}
-0.159834 (0.052822) with: {"learning_rate": 0.01, "n_estimators": 300}
-0.125159 (0.056987) with: {"learning_rate": 0.01, "n_estimators": 400}
-0.108292 (0.059112) with: {"learning_rate": 0.01, "n_estimators": 500}
-0.083225 (0.059937) with: {"learning_rate": 0.1, "n_estimators": 100}
-0.077744 (0.057482) with: {"learning_rate": 0.1, "n_estimators": 200}
-0.077754 (0.057472) with: {"learning_rate": 0.1, "n_estimators": 300}
-0.077754 (0.057472) with: {"learning_rate": 0.1, "n_estimators": 400}
-0.077754 (0.057472) with: {"learning_rate": 0.1, "n_estimators": 500}

Download Materials

What Users are saying..

profile image

Ameeruddin Mohammed

ETL (Abintio) developer at IBM
linkedin profile url

I come from a background in Marketing and Analytics and when I developed an interest in Machine Learning algorithms, I did multiple in-class courses from reputed institutions though I got good... Read More

Relevant Projects

Build a Similar Images Finder with Python, Keras, and Tensorflow
Build your own image similarity application using Python to search and find images of products that are similar to any given product. You will implement the K-Nearest Neighbor algorithm to find products with maximum similarity.

Loan Default Prediction Project using Explainable AI ML Models
Loan Default Prediction Project that employs sophisticated machine learning models, such as XGBoost and Random Forest and delves deep into the realm of Explainable AI, ensuring every prediction is transparent and understandable.

AWS Project to Build and Deploy LSTM Model with Sagemaker
In this AWS Sagemaker Project, you will learn to build a LSTM model on Sagemaker for sales forecasting while analyzing the impact of weather conditions on Sales.

Recommender System Machine Learning Project for Beginners-2
Recommender System Machine Learning Project for Beginners Part 2- Learn how to build a recommender system for market basket analysis using association rule mining.

A/B Testing Approach for Comparing Performance of ML Models
The objective of this project is to compare the performance of BERT and DistilBERT models for building an efficient Question and Answering system. Using A/B testing approach, we explore the effectiveness and efficiency of both models and determine which one is better suited for Q&A tasks.

Avocado Machine Learning Project Python for Price Prediction
In this ML Project, you will use the Avocado dataset to build a machine learning model to predict the average price of avocado which is continuous in nature based on region and varieties of avocado.

Build an Image Classifier for Plant Species Identification
In this machine learning project, we will use binary leaf images and extracted features, including shape, margin, and texture to accurately identify plant species using different benchmark classification techniques.

Hands-On Approach to Master PyTorch Tensors with Examples
In this deep learning project, you will learn how to perform various operations on the building block of PyTorch : Tensors.

Build a Text Classification Model with Attention Mechanism NLP
In this NLP Project, you will learn to build a multi class text classification model with attention mechanism.

Llama2 Project for MetaData Generation using FAISS and RAGs
In this LLM Llama2 Project, you will automate metadata generation using Llama2, RAGs, and AWS to reduce manual efforts.