Spark SQL Data Source Format #batch #realtime #dataengineering #funlearning #bigdata #pvdata смотреть онлайн

Spark SQL Data Source Format
#parquet
#json
#csv
#machinelearning
#singapore
#usa
#coding
#softwareengineering
#dataengineer
#highpaying

Learn about the various sources and formats of data that can be read and written using Spark SQL.
spark free course
Reading data into DataFrames
DataFrameReader
DataFrameWriter
Formats
Parquet
JSON
CSV
Other formats


Reading data into DataFrames#
Once data has been ingested, processed, and loaded into Spark SQL databases and tables, it can be read as DataFrames. An example is shown below:



Reading data into DataFrames
Once data has been ingested, processed, and loaded into Spark SQL databases and tables, it can be read as DataFrames. An example is shown below:


In the above example, we create the Spark SQL table movieData and then execute a Spark SQL query to return only the titles of the movies as a DataFrame.


DataFrameReader
We have touched upon DataFrameReader briefly in an earlier lesson. It is the core construct used for reading data from a source into a DataFrame. The pattern of stringing methods together is common in Spark and is also recommended when using DataFrameReader. The usage template occurs as follows:

DataFrameReader.format(args).option("key", "value").schema(args).load()
We can’t instantiate the DataFrameReader instance. Rather, it is available through a SparkSession instance e.g., SparkSession.read or SparkSession.readStream. The first API can be used to read static data sources, while the second one can be used to read a streaming source. In our previous examples, the spark variable represents the
SparkSession. DataFrameReader can read various data formats such as CSV, JSON, and Parquet, and in the case of static Parquet data files, the schema option can be skipped since the file comes embedded within the schema.


DataFrameWriter#
The counterpart to DataFrameReader is DataFrameWriter, which can be used to write the contents of a DataFrame to a built-in data source. The DataFrameWriter is accessible from an instance of DataFrame. The recommended usage pattern is:

DataFrameWriter.format(args).option(args).sortBy(args).saveAsTable(table)
Or:

DataFrameWriter.format(args).option(args).bucketBy(args).partitionBy(args).save(path)
To get an instance, we can invoke DataFrame.write or DataFrame.writeStream for static and streaming outputs, respectively.

Formats
Data can be read into and written out using DataFrames in the following formats:

Parquet
JSON
CSV
Avro
ORC
Images (machine learning and deep learning frameworks)
Binary Files
We have already seen the reading of data in CSV format in a DataFrame. Let’s see a few examples with the other formats.


Parquet#
For simplicity we’ll assume that we already have a DataFrame with the data we want to write out as a Parquet file.

https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html

https://github.com/vishnoiprem/scala-spark-educative


Apache Spark File Format Ecosystem