How to load data in batches with a DataLoader in MXNet

This recipe helps you load data in batches with a DataLoader in MXNet

Recipe Objective: How to load data in batches with a DataLoader in MXNet?

This recipe explains how to load data in batches with a DataLoader in MXNet.

Step 1: Importing library

Let us first import the necessary libraries. We'll import mxnet, os, tarfile, and multiprocessing from cpu_count.

import mxnet as mx
from mxnet import nd
from mxnet import autograd
from multiprocessing import cpu_count

Step 2: Dataset

We'll first fix the seed for reproducibility and generate random data set and pass them to ArrayDataset.

mxnet.random.seed(10)
a = mxnet.random.uniform(0,1,(5,15))
b = mxnet.random.uniform(0,1,(5,15))
data = mxnet.gluon.data.dataset.ArrayDataset(a, b)

Step 3: DataLoader

DataLoader in MXNet generates small batches of a sample from the dataset. It is better to pass data in the form of small packages rather than passing the whole dataset. This also gives an edge in multiprocessing.

load = mxnet.gluon.data.DataLoader(data, batch_size=3, num_workers=cpu_count())
for a_batch, b_batch in load:
    print("a has shape {}, and b has shape {}".format(a_batch.shape, b_batch.shape))

What Users are saying..

profile image

Gautam Vermani

Data Consultant at Confidential
linkedin profile url

Having worked in the field of Data Science, I wanted to explore how I can implement projects in other domains, So I thought of connecting with ProjectPro. A project that helped me absorb this topic... Read More

Relevant Projects

OpenCV Project for Beginners to Learn Computer Vision Basics
In this OpenCV project, you will learn computer vision basics and the fundamentals of OpenCV library using Python.

Build an AI Chatbot from Scratch using Keras Sequential Model
In this NLP Project, you will learn how to build an AI Chatbot from Scratch using Keras Sequential Model.

Deploy Transformer-BART Model on Paperspace Cloud
In this MLOps Project you will learn how to deploy a Tranaformer BART Model for Abstractive Text Summarization on Paperspace Private Cloud

Multilabel Classification Project for Predicting Shipment Modes
Multilabel Classification Project to build a machine learning model that predicts the appropriate mode of transport for each shipment, using a transport dataset with 2000 unique products. The project explores and compares four different approaches to multilabel classification, including naive independent models, classifier chains, natively multilabel models, and multilabel to multiclass approaches.

End-to-End Speech Emotion Recognition Project using ANN
Speech Emotion Recognition using RAVDESS Audio Dataset - Build an Artificial Neural Network Model to Classify Audio Data into various Emotions like Sad, Happy, Angry, and Neutral

Medical Image Segmentation Deep Learning Project
In this deep learning project, you will learn to implement Unet++ models for medical image segmentation to detect and classify colorectal polyps.

Build Classification Algorithms for Digital Transformation[Banking]
Implement a machine learning approach using various classification techniques in Python to examine the digitalisation process of bank customers.

MLOps Project for a Mask R-CNN on GCP using uWSGI Flask
MLOps on GCP - Solved end-to-end MLOps Project to deploy a Mask RCNN Model for Image Segmentation as a Web Application using uWSGI Flask, Docker, and TensorFlow.

Expedia Hotel Recommendations Data Science Project
In this data science project, you will contextualize customer data and predict the likelihood a customer will stay at 100 different hotel groups.

Learn How to Build a Logistic Regression Model in PyTorch
In this Machine Learning Project, you will learn how to build a simple logistic regression model in PyTorch for customer churn prediction.