How to programmatically specify a schema?

Official Website: http://bigdataelearning.com

When you don’t know the structure of the data ahead of time, you cannot create a case class. Or, if the structure of data is ever-changing, then also it is not possible to specify a static schema. For e.g. it may be the case that a new column gets added today (or) an existing column can get dropped the day after. In such cases, how do you specify the case class without knowing the schema.

In such instances, you can follow the following 4 steps to programmatically specify the schema, and create a dataframe.

Step - I
======
Remove the header row from the original Rdd and create rddWithNoHeader Rdd. Here I am applying top action to the rdd rdd1. This should pick the first row of the Rdd. I am also passing the row to the parallelize method of spark context. This creates the headerRdd.
picking the first header row using the top action and creating an Rdd out of it

To remove the header row from the original rdd, I am using subtract transformation. Rdd1 dot subtract off headerRdd. This gives us rddWithNoHeader. Removing the header row from the original rdd1, by using the subtract transformation

Step II
======
Now that we have rdd with no header, second step is to generate a schema from the top row. Here I am using the first action to extract the first row from the Rdd as a string. picking the first row as schemaString
Next I am splitting the string based on delimiter and generating the schema. The schema is generated by using StructType and StructField types. Generate the schema based on the schemaString

Step III
======
The 3rd step is to generate the row Rdd from the regular Rdd.
Here I am using map transformation on rddWithNoHeader to create Row Rdd. The elements are split based on comma as delimited to create array of values. I am applying another map transformation on top of it to create a row rdd. convert the regular rdd to row rdd

Step IV
=======
The final step is to create Dataframe by using sqlContext’s createDataFrame method. Here I am passing the rowRdd and schema as parameters. This gives us a new dataframe.

Creating dataframe using createDataFrame method. Using the rowRdd and schema.