Explain the creation of Dataframes in PySpark in Databricks

This recipe explains what the creation of Dataframes in PySpark in Databricks

Recipe Objective - Explain the creation of Dataframes in PySpark in Databricks?

The PySpark Dataframe is a distributed collection of the data organized into the named columns and is conceptually equivalent to the table in the relational database or the data frame in Python or R language. The Dataframes in PySpark can also be constructed from a wide array of the sources such as the structured data files, the tables in Apache Hive, External databases or the existing Resilient Distributed Datasets. Further, the DataFrame API(Application Programming Interface) is available in Java, Scala, Python and R. Also, the DataFrame is represented by the Dataset of Rows in Scala and Java. The DataFrame is the type alias of Dataset[Row] in the Scala API. The creation of the PySpark DataFrame is done using the "toDF()" and "createDataFrame()" methods and both this function takes different signatures to create the DataFrame from the existing RDD(Resilient Distributed Datasets), list, and DataFrame.

System Requirements

  • Python (3.0 version)
  • Apache Spark (3.1.1 version)

This recipe explains what is PySpark Dataframe is and how to create them in PySpark.

Build an Awesome Job Winning Data Engineering Projects Portfolio

Implementing the creation of Dataframes in Databricks in PySpark

# Importing packages
import pyspark
from pyspark.sql import SparkSession, Row
from pyspark.sql.types import StructType,StructField, StringType, IntegerType
Databricks-1

The Sparksession, Row, MapType, StringType, StructField, IntegerType are imported in the environment to create data frames in PySpark.

# Implementing the creation of Dataframes in Databricks in PySpark
spark = SparkSession.builder.appName('Creation Dataframe PySpark').getOrCreate()
columns = ["language","users_count"]
data = [("R", "30000"), ("Go", "200000"), ("Matlab", "2000")]
rdd = spark.sparkContext.parallelize(data)
# Using toDF() function
dataframeFromRDD1 = rdd.toDF()
dataframeFromRDD1.printSchema()
# Create Dataframe using Spark Session
dataframeFromRDD2 = spark.createDataFrame(rdd).toDF(*columns)
# Create Dataframe with the Schema
sample_data = [("Amit","","Gupta","36678","M",4000),
("Anita","Mathews","","40299","F",5000),
("Ram","","Aggarwal","42124","M",5000),
("Pooja","Anne","Goel","39298","F",5000),
("Geeta","Banuwala","Brown","","F",-2)
]
sample_schema = StructType([ \
StructField("firstname",StringType(),True), \
StructField("middlename",StringType(),True), \
StructField("lastname",StringType(),True), \
StructField("id", StringType(), True), \
StructField("gender", StringType(), True), \
StructField("salary", IntegerType(), True) \
])
dataframe = spark.createDataFrame(data = sample_data, schema = sample_schema)
dataframe.printSchema()
dataframe.show(truncate=False)
Databricks-1

Databricks-2
Databricks-3

The Spark Session is defined. The "columns", "data" and "rdd" are defined. Using toDF() function, the "dataframeFromRDD1" is created using the rdd. Further, the "dataframeFromRDD2" is created using the Spark Session. Also, the Dataframe is created using the "sample_data" and the "sample_schema" using the createDataFrame() function.

What Users are saying..

profile image

Abhinav Agarwal

Graduate Student at Northwestern University
linkedin profile url

I come from Northwestern University, which is ranked 9th in the US. Although the high-quality academics at school taught me all the basics I needed, obtaining practical experience was a challenge.... Read More

Relevant Projects

Build a Real-Time Dashboard with Spark, Grafana, and InfluxDB
Use Spark , Grafana, and InfluxDB to build a real-time e-commerce users analytics dashboard by consuming different events such as user clicks, orders, demographics

How to deal with slowly changing dimensions using snowflake?
Implement Slowly Changing Dimensions using Snowflake Method - Build Type 1 and Type 2 SCD in Snowflake using the Stream and Task Functionalities

Build a Data Pipeline with Azure Synapse and Spark Pool
In this Azure Project, you will learn to build a Data Pipeline in Azure using Azure Synapse Analytics, Azure Storage, Azure Synapse Spark Pool to perform data transformations on an Airline dataset and visualize the results in Power BI.

Snowflake Azure Project to build real-time Twitter feed dashboard
In this Snowflake Azure project, you will ingest generated Twitter feeds to Snowflake in near real-time to power an in-built dashboard utility for obtaining popularity feeds reports.

Python and MongoDB Project for Beginners with Source Code-Part 1
In this Python and MongoDB Project, you learn to do data analysis using PyMongo on MongoDB Atlas Cluster.

Yelp Data Processing using Spark and Hive Part 2
In this spark project, we will continue building the data warehouse from the previous project Yelp Data Processing Using Spark And Hive Part 1 and will do further data processing to develop diverse data products.

Orchestrate Redshift ETL using AWS Glue and Step Functions
ETL Orchestration on AWS - Use AWS Glue and Step Functions to fetch source data and glean faster analytical insights on Amazon Redshift Cluster

Build Classification and Clustering Models with PySpark and MLlib
In this PySpark Project, you will learn to implement pyspark classification and clustering model examples using Spark MLlib.

Learn Efficient Multi-Source Data Processing with Talend ETL
In this Talend ETL Project , you will create a multi-source ETL Pipeline to load data from multiple sources such as MySQL Database, Azure Database, and API to Snowflake cloud using Talend Jobs.

SQL Project for Data Analysis using Oracle Database-Part 6
In this SQL project, you will learn the basics of data wrangling with SQL to perform operations on missing data, unwanted features and duplicated records.