Big Data Project on Processing Unstructured Data using Spark

Big Data Project on Processing Unstructured Data using Spark

In this project, we will evaluate and demonstrate how to handle unstructured data using Spark.

Videos

Each project comes with 2-5 hours of micro-videos explaining the solution.

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

Project Experience

Add project experience to your Linkedin/Github profiles.

What will you learn

Giving unstructured data some structure
Programmatically creating data schema using Spark
Handling bad data
Revisiting Spark and Hive integration
Incremental updates in Spark
Automating your data pipeline

Project Description

Not all dataset comes structure. Or better put, there are more unstructured or semi-structured datasets that they are structured. And as a data engineer, we should at least give a good amount of structure or schema to data before it becomes useful for any downstream operation.

In this Hackerday session, we will evaluate and demonstrate how to handle rather unstructured data sets from the ginniemae.gov data disclosure history site. This dataset is a free text data that comes with a codebook describing the data. A lot does actually happen between the codebook and the data and we will see all in this sessions.

Ginnie Mae is a federally-owned corporation that helps to create and guarantee mortgage-backed securities in the US housing market. It is a lot more than that. See https://www.investopedia.com/terms/g/ginniemae.asp from more.

Similar Projects

The goal of this IoT project is to build an argument for generalized streaming architecture for reactive data ingestion based on a microservice architecture. 

In this hive project, you will design a data warehouse for e-commerce environments.

In this spark project, we will measure by how much NFP has triggered moves in past markets.

Curriculum For This Mini Project