How to build text preprocessing pipelines with Dask?
This recipe helps you build text preprocessing pipelines with Dask
Recipe Objective.
How to build text preprocessing pipelines with Dask?
`dask_ml.preprocessing` have same styled transformers of **scikit-learn** that we can use in Pipelines to perform different types of data transformations as the part of the model fitting process. These transformers works very nicely on dask collections (`dask.array, dask.dataframe`), NumPy arrays, or pandas dataframes.
Table of Contents
Step 1- Importing Libraries.
!apt install dask_ml
from dask_ml.preprocessing import Categorizer, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
import pandas as pd
import dask.dataframe as dd
Step 2- Creating a DataFrame.
We will create a dataframe and then divide it to x and y to fit them in the pipeline.
df = pd.DataFrame({"A": [1, 2, 3, 4, 5, 6], "B": ["a", "b", "c", "d", "e", "f"]})
x = dd.from_pandas(df, npartitions=2)
y = dd.from_pandas(pd.Series([0, 1, 1, 0]), npartitions=2)
Step 3- Creating a pipeline.
We will create a pipeline in which we process the data through Categorizer, OneHotEncoder, LogisticRegression.
pipe = make_pipeline(
Categorizer(),
OneHotEncoder(),
LogisticRegression(solver='lbfgs')
)
pipe.fit(x, y)
```
Pipeline(steps=[('categorizer', Categorizer()),
('onehotencoder', OneHotEncoder()),
('logisticregression', LogisticRegression())]) ```
What Users are saying..
Ray han
I think that they are fantastic. I attended Yale and Stanford and have worked at Honeywell,Oracle, and Arthur Andersen(Accenture) in the US. I have taken Big Data and Hadoop,NoSQL, Spark, Hadoop... Read More