What are lazy functions and how does dask deal with them?
This recipe explains what are lazy functions and how does dask deal with them
Recipe Objective
What are lazy functions? How does dask deal with them.
In python we have something called lazy functions specially for big DataFrames we call it lazy execution of code.
Sometimes problems don't fit in the single code, or the RAM could not hold the long execution of code, sometimes dask.arrays or dask.dataframe fails to manage the long Datasets. To overcome those problems we can use dask.delayed interface in this the users can parallelize the custom algorithms.It allows users to create graphs directly with a light annotation of normal python code.
Access Product Recommendation System Project with Source Code
Step 1- Importing Libraries.
import dask
Step 2- Creating a Dask Delayed code for sample.
We will define some sample functions and then we will apply the Delay function after combining all the functions.
def increase(x):
return x + 2
def triple(x):
return x * 3
def divide(x,y):
return y/x
def add(x, y, z):
return x + y +z
Step 3- Creating a sample list.
Now we will pass the list from every defined function and compute the final data.
data = [5, 10, 15, 20, 25, 30, 35, 40]
y=100
output = []
for x in data:
a = increase(x)
b = triple(x)
c = divide(x,y)
d = add(a, b, c)
output.append(d)
total = dask.delayed(sum)(output)
total
What Users are saying..
Ed Godalle
I am the Director of Data Analytics with over 10+ years of IT experience. I have a background in SQL, Python, and Big Data working with Accenture, IBM, and Infosys. I am looking to enhance my skills... Read More