R Tutorial: R packages for parallel computing смотреть онлайн

Want to learn more? Take the full course at https://learn.datacamp.com/courses/parallel-programming-in-r at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work.

---

In this lesson, we will talk about a few R packages that support parallel computing.

The package we will talk about most in this course is the R-core package parallel. The advantage of using solely the parallel package is that your code does not need to be dependent on any other packages which sometimes can cause problems. However, the power of R comes from user-contributed packages, after all, so using other packages can make things simpler and easier for you.

In the previous lesson, we talked about the map-reduce paradigm implemented for example in Hadoop and Spark, or more generally parallel approach to big data. If you want to experiment with that approach, you may look at the R package sparklyr which offers an interface to the Apache Spark engine and the iotools package presented in the DataCamp course Scalable Data Processing in R.

I also recommend you check out the suite of packages pbdR which stands for Programming with Big Data in R.

In this course, we will focus on R packages that provide support for embarrassingly parallel applications using the master-worker model.

The two packages that we will talk about in more detail, in addition to the parallel package, are foreach and future.apply.

The snow package is the pioneer of this approach, and the core package parallel is mostly a reimplementation of snow. However, not all snow functionality has been ported over to parallel. snowFT is an extension to snow that adds important features, such as reproducibility and ease of use. Adding simplicity to snow is also the main focus of the package snowfall. However, these three packages may be a little outdated.

That's not the case with the package future which is under active development. It provides an abstraction layer or a unified API for sequential and parallel processing. The future.apply package which we will talk about in this course is an implementation of the apply-type functions used in the future.

You should be familiar with the basics of the parallel package since it was covered in the Writing Efficient R code course.

You can use the function detectCores() to find out how many cores your computer has. If you are interested only in the physical cores, set the argument logical to FALSE.

Function makeCluster() creates a cluster of nodes, or a cluster of workers. Here we created as many workers as we have physical cores. The current R session serves as the master process, while each worker is a separate R process.

The workhorse of the parallel package is the function clusterApply(). Its first argument is the cluster object, here cl. The second argument x is a sequence whose length determines how many times the function fun, which is the third argument, is going to be evaluated. The evaluation is spread across the workers, and the elements of the x sequence are passed as the first argument of the function fun().

When a cluster is not needed anymore, it is closed using the function stopCluster().

In our example, where ncores is four, and therefore x in clusterApply() is a sequence from four to one, the first worker gets the instruction from the master to evaluate rnorm(4), the second worker rnorm(3), third worker rnorm(2) and finally, the fourth worker gets the number one as its first argument. The master then collects the results, which in this case are 10 random numbers.

Time to put this into practice.