R tutorial: Cleaning and preprocessing text

Learn more about text mining with R: https://www.datacamp.com/courses/intro-to-text-mining-bag-of-words

Now that you have a corpus, you have to take it from the unorganized raw state and start to clean it up. We will focus on some common preprocessing functions. But before we actually apply them to the corpus, let’s learn what each one does because you don’t always apply the same ones for all your analyses.

Base R has a function tolower. It makes all the characters in a string lowercase. This is helpful for term aggregation but can be harmful if you are trying to identify proper nouns like cities.

The removePunctuation function...well it removes punctuation. This can be especially helpful in social media but can be harmful if you are trying to find emoticons made of punctuation marks like a smiley face.

Depending on your analysis you may want to remove numbers. Obviously don’t do this if you are trying to text mine quantities or currency amounts but removeNumbers may be useful sometimes.

The stripWhitespace function is also very useful. Sometimes text has extra tabbed whitespace or extra lines. This simply removes it.

A very important function from tm is removeWords. You can probably guess that a lot of words like "the" and "of" are not very interesting, so may need to be removed.

All of these transformations are applied to the corpus using the tm_map function. This text mining function is an interface to transform your corpus through a mapping to the corpus content. You see here the tm_map takes a corpus, then one of the preprocessing functions like removeNumbers or removePunctuation to transform the corpus. If the transforming function is not from the tm library it has to be wrapped in the content_transformer function. Doing this tells tm_map to import the function and use it on the content of the corpus.

The stemDocument function uses an algorithm to segment words to their base. In this example, you can see "complicatedly", "complicated" and "complication" all get stemmed to "complic". This definitely helps aggregate terms. The problem is that you are often left with tokens that are not words! So you have to take an additional step to complete the base tokens. The stemCompletion function takes as arguments the stemmed words and a dictionary of complete words. In this example, the dictionary is only "complicate", but you can see how all three words were unified to "complicate". You can even use a corpus as your completion dictionary as shown here.

There is another whole group of preprocessing functions from the qdap package which can complement these nicely. In the exercises, you will have the opportunity to work with both tm and qdap preprocessing functions, then apply them to a corpus.