How to load data in batches with a DataLoader in MXNet

This recipe helps you load data in batches with a DataLoader in MXNet

Recipe Objective: How to load data in batches with a DataLoader in MXNet?

This recipe explains how to load data in batches with a DataLoader in MXNet.

Step 1: Importing library

Let us first import the necessary libraries. We'll import mxnet, os, tarfile, and multiprocessing from cpu_count.

import mxnet as mx
from mxnet import nd
from mxnet import autograd
from multiprocessing import cpu_count

Step 2: Dataset

We'll first fix the seed for reproducibility and generate random data set and pass them to ArrayDataset.

mxnet.random.seed(10)
a = mxnet.random.uniform(0,1,(5,15))
b = mxnet.random.uniform(0,1,(5,15))
data = mxnet.gluon.data.dataset.ArrayDataset(a, b)

Step 3: DataLoader

DataLoader in MXNet generates small batches of a sample from the dataset. It is better to pass data in the form of small packages rather than passing the whole dataset. This also gives an edge in multiprocessing.

load = mxnet.gluon.data.DataLoader(data, batch_size=3, num_workers=cpu_count())
for a_batch, b_batch in load:
    print("a has shape {}, and b has shape {}".format(a_batch.shape, b_batch.shape))

What Users are saying..

profile image

Ed Godalle

Director Data Analytics at EY / EY Tech
linkedin profile url

I am the Director of Data Analytics with over 10+ years of IT experience. I have a background in SQL, Python, and Big Data working with Accenture, IBM, and Infosys. I am looking to enhance my skills... Read More

Relevant Projects

Build CNN Image Classification Models for Real Time Prediction
Image Classification Project to build a CNN model in Python that can classify images into social security cards, driving licenses, and other key identity information.

Avocado Machine Learning Project Python for Price Prediction
In this ML Project, you will use the Avocado dataset to build a machine learning model to predict the average price of avocado which is continuous in nature based on region and varieties of avocado.

Hands-On Approach to Master PyTorch Tensors with Examples
In this deep learning project, you will learn how to perform various operations on the building block of PyTorch : Tensors.

MLOps AWS Project on Topic Modeling using Gunicorn Flask
In this project we will see the end-to-end machine learning development process to design, build and manage reproducible, testable, and evolvable machine learning models by using AWS

Learn How to Build PyTorch Neural Networks from Scratch
In this deep learning project, you will learn how to build PyTorch neural networks from scratch.

Build Classification Algorithms for Digital Transformation[Banking]
Implement a machine learning approach using various classification techniques in Python to examine the digitalisation process of bank customers.

Build a Review Classification Model using Gated Recurrent Unit
In this Machine Learning project, you will build a classification model in python to classify the reviews of an app on a scale of 1 to 5 using Gated Recurrent Unit.

Recommender System Machine Learning Project for Beginners-4
Collaborative Filtering Recommender System Project - Comparison of different model based and memory based methods to build recommendation system using collaborative filtering.

Ola Bike Rides Request Demand Forecast
Given big data at taxi service (ride-hailing) i.e. OLA, you will learn multi-step time series forecasting and clustering with Mini-Batch K-means Algorithm on geospatial data to predict future ride requests for a particular region at a given time.

Llama2 Project for MetaData Generation using FAISS and RAGs
In this LLM Llama2 Project, you will automate metadata generation using Llama2, RAGs, and AWS to reduce manual efforts.