How to convert Categorical variables into Numerical Variables using ColumnTransformer in python

This recipe is a short example on how to convert categorical variables into numerical variables using ColumnTransformer in python

Recipe Objective

When we talk about categorical variables, we are talking about non numerical values like Strings or Text. For example, a city name, a store name - for that matter anything that is not a number is a categorical variable. For us humans, it is very easy to interpret and understand text - but for machines it is not the case.

In Machine Learning, all the algorithms in the sklearn library cannot handle categorical variables so before we give the data to the algorithm for training and predicting - we have to convert it to numbers.

So this recipe is a short example on how to convert categorical variables into numerical variables. Let's get started.

Step 1 - Import the library - ColumnTransformer

import numpy as np import pandas as pd from sklearn.preprocessing import OneHotEncoder from sklearn.compose import ColumnTransformer

Let's pause and look at these imports. Numpy and Pandas are the usual ones. If you've looked at the older recipes , LabelEncoder and OneHotEncoder was typically used to convert categorical variables. But here we are using ColumnTransformer - why?

In the newer versions of sklearn starting from 0.20 - they have introduced ColumnTransformer. Prior to this, doing this conversion was a 2 step process. First we have to label encode the data and then one hot encode it. But now with ColumnTransformer, they have combined this - so you can do the entire conversion with just one line of code as shown in Step 3 below.

Step 2 - Setup the Data

dataset = {'research': [15000, 90000, 250000, 175000, 88000, 210000], 'marketing': [5000, 25000, 31000, 44000, 19700, 21111], 'city': ['Texas', 'Delaware', 'Florida', 'Texas', 'Delaware','Florida'], 'profit': [9999, 5555, 3333, 4444, 1111, 2222] df = pd.DataFrame(dataset)

Let us create a simple dataset and convert it to a dataframe. This sample data shows how much a company spends on research and marketing. We will add the state where the company is present. The last variable is the profit that company makes. So in a real example, we will be predicting the profit. But in this recipe, we will not be going into that.

Now our dataset is ready

Step 3 - Create an object of ColumnTransformer class

Before we do that, let's look at the important parameters that we need to pass.


1) transformer

  • name: this is just a name that we pass to the transformer
  • transformer: here we need to pass an estimator that supports fit and transform. Since we want to encode the data, we will be passing OneHotEncoder here.
  • columns: the index of columns that contain the categorical values that we want to convert

2) remainder
This parameter basically tells the transformer what to do with the remaining columns other than the ones we have mentioned above for converting. The values can be "drop" or "passthrough". The default value is drop which means only the transformed columns will be returned by the transformer and the remaining columns will be dropped. But if we want the transformer to pass them - we have to use the value "passthrough"

Now that we understand, let's create the object

columnTransformer = ColumnTransformer([('encoder', OneHotEncoder(), [2])], remainder='passthrough')

  • As you can see, we have named the transformer as "encoder"
  • For the estimator, we are passing an object of OneHotEncoder
  • And for columns, we can see from our dataset that the 3rd column is the city column that we want to transform. Since the index start at 0, we are passing value 2 here
  • Lastly for remainder we are passing "passthrough" as we want the transformer to return the remaining columns

Step 4 - Convert the categorical data with one line of code

Now that we have got the ColumnTransformer constructor ready, we just have to call the fit_transform method and pass the dataset to it to do the conversion

df = np.array(columnTransformer.fit_transform(df), dtype = np.str)

Step 5 - Lets look at our dataset now

Once we run the above code snippet, we will see that all States have been converted to numbers and added to the beginning.

For example :
Texas has been represented by 0,0,1
Delaware by 1,0, 0
Florida by 0,1,0.

print(df)

[['0.0' '0.0' '1.0' '15000.0' '5000.0' '9999.0']
 ['1.0' '0.0' '0.0' '90000.0' '25000.0' '5555.0']
 ['0.0' '1.0' '0.0' '250000.0' '31000.0' '3333.0']
 ['0.0' '0.0' '1.0' '175000.0' '44000.0' '4444.0']
 ['1.0' '0.0' '0.0' '88000.0' '19700.0' '1111.0']
 ['0.0' '1.0' '0.0' '210000.0' '21111.0' '2222.0']]

What Users are saying..

profile image

Abhinav Agarwal

Graduate Student at Northwestern University
linkedin profile url

I come from Northwestern University, which is ranked 9th in the US. Although the high-quality academics at school taught me all the basics I needed, obtaining practical experience was a challenge.... Read More

Relevant Projects

A/B Testing Approach for Comparing Performance of ML Models
The objective of this project is to compare the performance of BERT and DistilBERT models for building an efficient Question and Answering system. Using A/B testing approach, we explore the effectiveness and efficiency of both models and determine which one is better suited for Q&A tasks.

Build a Collaborative Filtering Recommender System in Python
Use the Amazon Reviews/Ratings dataset of 2 Million records to build a recommender system using memory-based collaborative filtering in Python.

Learn How to Build a Linear Regression Model in PyTorch
In this Machine Learning Project, you will learn how to build a simple linear regression model in PyTorch to predict the number of days subscribed.

Build a Face Recognition System in Python using FaceNet
In this deep learning project, you will build your own face recognition system in Python using OpenCV and FaceNet by extracting features from an image of a person's face.

MLOps AWS Project on Topic Modeling using Gunicorn Flask
In this project we will see the end-to-end machine learning development process to design, build and manage reproducible, testable, and evolvable machine learning models by using AWS

Build a Graph Based Recommendation System in Python -Part 1
Python Recommender Systems Project - Learn to build a graph based recommendation system in eCommerce to recommend products.

Word2Vec and FastText Word Embedding with Gensim in Python
In this NLP Project, you will learn how to use the popular topic modelling library Gensim for implementing two state-of-the-art word embedding methods Word2Vec and FastText models.

Deep Learning Project for Time Series Forecasting in Python
Deep Learning for Time Series Forecasting in Python -A Hands-On Approach to Build Deep Learning Models (MLP, CNN, LSTM, and a Hybrid Model CNN-LSTM) on Time Series Data.

Mastering A/B Testing: A Practical Guide for Production
In this A/B Testing for Machine Learning Project, you will gain hands-on experience in conducting A/B tests, analyzing statistical significance, and understanding the challenges of building a solution for A/B testing in a production environment.

AWS MLOps Project for ARCH and GARCH Time Series Models
Build and deploy ARCH and GARCH time series forecasting models in Python on AWS .