Machine Learning: Text Preprocessing and Vectorization

Data Science course by www.6benches.in
-------------------------------
Below topics covered in this Text preprocessing and vectorization lecture:
1)Process of converting text/unstructured data to structured data is vectorization.
2) Three types of vectrization:
2a) bag of words model
2b) count vectorizer
2c) tf-idf vectorizer

3) from sklearn.datasets import fetch_20newsgroups
twenty_train = fetch_20newsgroups(subset = 'train', shuffle = true)
twenty_train.target_names #display target categories
set(twenty_train.target) #display target category labelled
len(twenty_train.data) #number of input records

4)from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
text = ["the quick brown fox jumped over lazy dog"]
vectorizer.fit(text) #tokenize and build vocabulary
print(vectorizer.vocabulary_) #summarize
vector = vectorizer.transform(text)#encode document
print(vector.shape)
print(vector.toarry())
print(type(vector))

5) Bag of words: gives importance to important words and less importance of non-important words
a)collect data
b)Design the vocabulary
c)create document vectors

6)tfidf acronym for Term frequency and inverse document frequency, is used often over the other two methods because of their limitation. Term frequency is calculated *within that* document and inverse document frequency calculates the frequency across *all* the documents and downscales it.
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer()
vector = vectorizer.fit_transform(text)
x = vector.toarray() #share remains same but converts csr sparse matrix to array
x[0] #display the probability(meaningful representation of words into numbers) of words in 0 document

Feed the above structured data(text to numbers) to ML models to classify text.