R Tutorial: Exploring the MNIST dataset

Want to learn more? Take the full course at https://learn.datacamp.com/courses/advanced-dimensionality-reduction-in-r at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work.

---
Hello, this is Federico Castanedo and I will explain to you how to apply advanced dimensionality reduction techniques to solve exciting problems. I got a Phd in Artificial Intelligence and have many years of data science experience in academia, start-ups, and large corporations.

Before we start, please remember this is an advanced course and we expect you have taken the previous course on Dimensionality Reduction.

Let's begin with a brief introduction of the dimensionality reduction techniques that we will learn and then we will explore the MNIST dataset.

In this course, we will learn how to apply two state-of-the-art dimensionality reduction techniques: t-SNE and Generalized Low Rank models. t-Distributed Stochastic Neighbor Embedding or t-SNE, is an algorithm that performs non-linear dimensionality reduction and we will explore how to use it in predictive models. On the other hand, we will also review GLRM, which is a parallelized optimisation algorithm that can be used with numerical and categorical variables and allows to impute missing values.

Dimensionality reduction techniques are based on unsupervised machine learning algorithms and their application offers several advantages: it provides a way of doing feature selection; it compresses high dimensional data into a few important features; it saves memory and speeds up building machine learning models; it allows the visualisation of high dimensional datasets; and in the case of GLRM it also imputes missing data.

In this course we will learn how to apply these dimensionality reduction techniques to exploit the mentioned advantages, using interesting datasets like the MNIST, a credit card fraud dataset from Kaggle and the fashion version of MNIST released by Zalando.

Before we start reducing data dimensionality, let's have a look at the MNIST dataset.

The MNIST dataset is a very well-known dataset used to evaluate the performance of Machine Learning models. It consists of 70.000 images of handwritten digits ranging from 0 to 9.
Each image is 28 pixels in height and 28 pixels in width, which makes a total of 784 pixels. Every pixel has a single value associated with it, indicating its lightness or darkness, which is an integer between 0 and 255. In this image, you can see an example of the number 3. Let's look at some more digit samples.

Here you can see more examples of handwritten digits from 0 to 9. It is clear that as humans we have several ways to write each digit and a machine must be able to take all of them into account. As we will see this is not an easy task.

Let's have a look at the first 6 records and columns.

As another example, we are showing the values of pixels 400 to 405 for the first record.
Note, that we select columns 402 to 407 because the first column is the label and the pixels start at 0.
Remember that each pixel is an integer between 0 and 255 that indicates its lightness or darkness.

We can expect that same digits will have similar values in each pixel, so one assumption is to compute the pixel statistics of the same digits. For instance, here we show the statistics of digits 1 and 0 for the pixel 408. As you can see, they have extremely different values for that pixel.

Let’s get started with the MNIST dataset and do some data exploration.

#R #RTutorial #DataCamp #Advanced #Dimensionality #Reduction #MNIST #dataset