What does model calibration mean in ML in python

This recipe explains what does model calibration mean in ML in python

Recipe Objective

Generally, for any classification problem, we predict the class value that has the highest probability of being the true class label. However, sometimes, we want to predict the probabilities of a data instance belonging to each class label. This type of problems can easily be handled by calibration curve. It support models with 0 and 1 value only.

So this recipe is a short example on what does caliberation mean. Let's get started.

Access House Price Prediction Project using Machine Learning with Source Code

Step 1 - Import the library

from sklearn import datasets from sklearn.model_selection import train_test_split from sklearn.datasets import load_breast_cancer from sklearn.tree import DecisionTreeClassifier from sklearn.calibration import calibration_curve import matplotlib.pyplot as plt

Let's pause and look at these imports. We have exported train_test_split which helps in randomly breaking the datset in two parts. Here sklearn.dataset is used to import one classification based model dataset. Also, we have exported calibration_curve to calibrate our model.

Step 2 - Setup the Data

X,y=load_iris(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)

Here, we have used load_iris function to import our dataset in two list form (X and y) and therefore kept return_X_y to be True. Further with have broken down the dataset into 2 parts, train and test with ratio 3:4.

Now our dataset is ready.

Step 3 - Building the model

model =DecisionTreeClassifier(criterion ='entropy', max_features = 2)

We have simply built a classification model with =DecisionTreeClassifier with criterion as entropy and max_feature to be 2.

Step 4 - Fit the model and predict for test set

model.fit(X_train, y_train) y_pred= model.predict(X_test)

Here we have simply fit used fit function to fit our model on X_train and y_train. Now, we are predicting the values of X_test using our built model.

Step 5 - Calibrating our model

x, y = calibration_curve(y_test, y_pred, n_bins = 10, normalize = True)

Now we are calibrating our predicted value to the actual value. n_bins refers for number of bins to discretize the [0, 1] interval. Also, we are normalizing y_pred in the [0,1] interval.

Step 5 - Plotting results

plt.plot([0, 1], [0, 1], linestyle = '--', label = 'Ideally Calibrated') plt.plot(y, x, marker = '.', label = 'Decision Tree Classifier') plt.xlabel('Average Predicted Probability in each bin') plt.ylabel('Ratio of positives') plt.legend() plt.show()

Here, first we have plotted the Ideally calibrated curve which will a straight line between 0 and 1. Now, we plot our calibrated curve of this particular model. he x-axis represents the average predicted probability in each bin. The y-axis is the ratio of positives (the proportion of positive predictions).

Step 7 - Lets look at our dataset now

Once we run the above code snippet, we will see:

Scroll down to the ipython file to visualize the results.

Clearly, the model built is highly efficient on any unknown set.

What Users are saying..

profile image

Jingwei Li

Graduate Research assistance at Stony Brook University
linkedin profile url

ProjectPro is an awesome platform that helps me learn much hands-on industrial experience with a step-by-step walkthrough of projects. There are two primary paths to learn: Data Science and Big Data.... Read More

Relevant Projects

Learn to Build a Siamese Neural Network for Image Similarity
In this Deep Learning Project, you will learn how to build a siamese neural network with Keras and Tensorflow for Image Similarity.

Deploy Transformer-BART Model on Paperspace Cloud
In this MLOps Project you will learn how to deploy a Tranaformer BART Model for Abstractive Text Summarization on Paperspace Private Cloud

Personalized Medicine: Redefining Cancer Treatment
In this Personalized Medicine Machine Learning Project you will learn to classify genetic mutations on the basis of medical literature into 9 classes.

Digit Recognition using CNN for MNIST Dataset in Python
In this deep learning project, you will build a convolutional neural network using MNIST dataset for handwritten digit recognition.

Build a Multi Touch Attribution Machine Learning Model in Python
Identifying the ROI on marketing campaigns is an essential KPI for any business. In this ML project, you will learn to build a Multi Touch Attribution Model in Python to identify the ROI of various marketing efforts and their impact on conversions or sales..

Deploying Machine Learning Models with Flask for Beginners
In this MLOps on GCP project you will learn to deploy a sales forecasting ML Model using Flask.

Learn Object Tracking (SOT, MOT) using OpenCV and Python
Get Started with Object Tracking using OpenCV and Python - Learn to implement Multiple Instance Learning Tracker (MIL) algorithm, Generic Object Tracking Using Regression Networks Tracker (GOTURN) algorithm, Kernelized Correlation Filters Tracker (KCF) algorithm, Tracking, Learning, Detection Tracker (TLD) algorithm for single and multiple object tracking from various video clips.

Loan Eligibility Prediction Project using Machine learning on GCP
Loan Eligibility Prediction Project - Use SQL and Python to build a predictive model on GCP to determine whether an application requesting loan is eligible or not.

Build a Review Classification Model using Gated Recurrent Unit
In this Machine Learning project, you will build a classification model in python to classify the reviews of an app on a scale of 1 to 5 using Gated Recurrent Unit.

Build an End-to-End AWS SageMaker Classification Model
MLOps on AWS SageMaker -Learn to Build an End-to-End Classification Model on SageMaker to predict a patient’s cause of death.