10 - Apache Spark for Java Developers - RDD Operations

@backstreetbrogrammer

--------------------------------------------------------------------------------
Chapter 01 - Apache Spark for Java Developers - RDD Operations
--------------------------------------------------------------------------------
RDDs support 2 kinds of operations:
1. Transformation – Spark RDD transformation is a function that produces new RDD from the existing RDDs. The transformer takes RDD as input and produces one or more RDD as output. Transformations are lazy in nature i.e., they get execute when we call an action.

We can divide transformations into two types as below:
- Narrow Transformations : We apply narrow transformations on to a single partition of the parent RDD to generate a new RDD as data required to process the RDD is available on a single partition of the parent RDD.
The examples for narrow transformations are:
map()
filter()
flatMap()
partition()
mapPartitions()

- Wide Transformations: We apply the wide transformation on multiple partitions to generate a new RDD. The data required to process the RDD is available on the multiple partitions of the parent RDD.
The examples for wide transformations are:
reduceBy()
union()

2. Action – transformations create RDDs from each other, but when we want to work with the actual data set, at that point action is performed. Thus, Actions are Spark RDD operations that give non-RDD values. The values of action are stored to drivers or to the external storage system.
Few of the actions include:
collect()
count()
take()
first()

An action is one of the ways of sending data from Executor to the driver.

Executors are agents that are responsible for executing a task. While the driver is a JVM process that coordinates workers and execution of the task.

Github: https://github.com/backstreetbrogrammer/11_JavaSpark

- Apache Spark for Java Developers Playlist: https://youtube.com/playlist?list=PLQDzPczdXrTgqEc0uomGYDS0SFu7qY3g3
- Java Serialization Playlist: https://youtube.com/playlist?list=PLQDzPczdXrTjGLkI8kd2d0bfNHx0v7dTy
- Dynamic Programming Playlist: https://youtube.com/playlist?list=PLQDzPczdXrTilRgmNUQ632OrD-wfRLUR_

#java #javadevelopers #javaprogramming #apachespark #spark