How to use for loop with dask for parallel processing?
This recipe helps you use for loop with dask for parallel processing
Recipe Objective
How to use for loop with dask for parallel processing.
We will transform the function inc to be used parallely with the help of dask delayed, and we will show you the visualizaion of the for loop to understand parallel working of it.
Table of Contents
Step 1- Importing Libraries.
from dask import delayed
Step 2- Defining a function.
We will define a incremental function which will increase the value by 2 in a for loop.
def inc(x):
return x + 2
Step 3- Applying delayed.
We will apply dask delayed in the for loop to run inc function parallely.
data = [1, 2, 3, 4, 5]
results = []
for x in data:
y = delayed(inc)(x)
results.append(y)
Step 4- Displaying results.
We will print the results of calculating addition and final computing.
addition = delayed(sum)(results)
print("Before addition:", addition)
calculate = addition.compute()
print("After calculating :", calculate)
Step 5- Visualizing
Visualizing the whole addition process that was done in parallel.
addition.visualize()
What Users are saying..
Savvy Sahai
As a student looking to break into the field of data engineering and data science, one can get really confused as to which path to take. Very few ways to do it are Google, YouTube, etc. I was one of... Read More