24 - Create Empty RDD using parallelize method - Code Demo смотреть онлайн

@backstreetbrogrammer

--------------------------------------------------------------------------------
Chapter 04 - Create Empty RDD using parallelize method - Code Demo
--------------------------------------------------------------------------------
Parallelized collections are created by calling JavaSparkContext’s parallelize() method on an existing Collection in the driver program. The elements of the collection are copied to form a RDD that can be operated on in parallel.

final var data = List.of(1, 2, 3, 4, 5, 6, 7, 8);
final var myRdd = sc.parallelize(data);

RDD created myRdd can be operated on in parallel. These operations can be to reduce, map, etc.

final var max = myRdd.reduce(Integer::max);
final var min = myRdd.reduce(Integer::min);
final var sum = myRdd.reduce(Integer::sum);

One important parameter for parallel collections is the number of partitions to cut the dataset into. Spark will run one task for each partition of the cluster.

We may want 2-4 partitions for each CPU in the cluster. Spark tries to set the number of partitions automatically based on our cluster.

We can get the number of partitions being used by calling this method:
myRdd.getNumPartitions()

However, we can also set it manually by passing it as a second parameter to parallelize() method.
sc.parallelize(data, 10)

We can also create Empty RDDs by calling emptyRDD() method on JavaSparkContext object.
final var emptyRDD = sparkContext.emptyRDD();


Github: https://github.com/backstreetbrogrammer/11_JavaSpark

- Apache Spark for Java Developers Playlist: https://youtube.com/playlist?list=PLQDzPczdXrTgqEc0uomGYDS0SFu7qY3g3
- Java Serialization Playlist: https://youtube.com/playlist?list=PLQDzPczdXrTjGLkI8kd2d0bfNHx0v7dTy
- Dynamic Programming Playlist: https://youtube.com/playlist?list=PLQDzPczdXrTilRgmNUQ632OrD-wfRLUR_

#java #javadevelopers #javaprogramming #apachespark #spark