Python Tutorial: Class distribution смотреть онлайн

Want to learn more? Take the full course at https://learn.datacamp.com/courses/ob... at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work.

---
One of the most necessary steps for preprocessing, which you should be familiar with if you've taken other courses on Python and machine learning, is splitting up your data into training and test sets. We do this to avoid the issue of overfitting. If we train a model on our entire set of data, we won't have any way to test and validate our model because the model will essentially know the dataset by heart. Holding out a test set allows us to preserve some data the model hasn't seen yet.

Just to review, this is how you split up your dataset in scikit learn using the train_test_split function. This should look familiar to you. The function shuffles up your dataset and then randomly splits it. By default, the function will split 75% of the data into the training set and 25% into the test set. In many scenarios, the default splitting parameters will work well. However, if your labels have an uneven distribution, your test and training sets might not be representative samples of your dataset and could bias the model you're trying to train. For example, if you look at the example training and test datasets on this slide, you can see that the training set has only samples labeled n, while there is a y label in the test set.

A good technique for sampling more accurately when you have imbalanced classes is stratified sampling, which is a way of sampling that takes into account the distribution of classes or features in your dataset. So for example, let's say we had a dataset with 100 samples, 80 of which are class 1 and 20 of which are class 2. We want the class distribution in both our training set and our test set to reflect this, so in both our training and test sets, we'd want 80% of our sample to be class 1 and 20% to be class 2, which means we'd want 60 class 1 samples and 15 class 2 samples in our training set of 75 samples. In our test set of 25 samples, we want to have 20 class 1 samples and 5 of class 2. This is on par with the distribution of classes in the original dataset.

There's a really easy way to do this in scikit learn using the train test split function. The function comes with a stratify parameter, and to stratify according to class labels, just pass in your y dataset to that parameter. So here we have our 100 labels, 80 are class1 and 20 are class 2. let's run train_test_split, and pass the y labels dataset into that stratify parameter.

If we check the distribution of classes for our training and test labels, you can see the distribution of classes is in accordance with the original y class distribution.

Now it's your turn to do some stratified sampling!

#Python #PythonTutorial #DataCamp #Preprocessing #MachineLearning #Class #distribution