Python for Machine Learning - Part 27 - Detecting Outliers using Elliptic Envelope

Python for Machine Learning - Part 27 - Detecting Outliers using Elliptic Envelope

Detecting Outliers using Elliptic Envelope

Github link - https://github.com/technologycult/PythonForMachineLearning/tree/master/Part27

Many applications require being able to decide whether a new observation belongs to the same distribution as existing observations (it is an inlier), or should be considered as different (it is an outlier). Often, this ability is used to clean real data sets. Two important distinctions must be made:
outlier detection
The training data contains outliers which are defined as observations that are far from the others. Outlier detection estimators thus try to fit the regions where the training data is the most concentrated, ignoring the deviant observations.
novelty detection
The training data is not polluted by outliers and we are interested in detecting whether a new observation is an outlier. In this context an outlier is also called a novelty.
Outlier detection and novelty detection are both used for anomaly detection, where one is interested in detecting abnormal or unusual observations. Outlier detection is then also known as unsupervised anomaly detection and novelty detection as semi-supervised anomaly detection. In the context of outlier detection, the outliers/anomalies cannot form a dense cluster as available estimators assume that the outliers/anomalies are located in low density regions. On the contrary, in the context of novelty detection, novelties/anomalies can form a dense cluster as long as they are in a low density region of the training data, considered as normal in this context.

Code Begins Here
'''

Topic to be covered - Detect Outliers using Elliptic Envelopes

'''

Code Starts Here
==============

import pandas as pd
import numpy as np

from sklearn.covariance import EllipticEnvelope

dataset = pd.read_csv('Salary.csv')


X = np.array([[100,100],
[1,1],
[2,4],
[5,6],
[6,8]])

outlier = EllipticEnvelope(contamination=0.1)

outlier.fit(X)
prediction1 = outlier.predict(X)
print(prediction1)

--------------------------------------

features = dataset.iloc[:,[1,2]].values

outlier1 = EllipticEnvelope(contamination=0.1)

outlier1.fit(features)
prediction2 = outlier1.predict(features)

print(prediction2)

dataset['outliers'] = prediction2


All Playlist of this youtube channel
====================================

1. Data Preprocessing in Machine Learning
https://www.youtube.com/playlist?list=PLE-8p-CwnFPuOjFcbnXLFvSQaHFK3ymUW

2. Confusion Matrix in Machine Learning, ML, AI
https://www.youtube.com/playlist?list=PLE-8p-CwnFPvXzvsEcgb0IZtNsw_0vUzr

3. Anaconda, Python Installation, Spyder, Jupyter Notebook, PyCharm, Graphviz
https://www.youtube.com/playlist?list=PLE-8p-CwnFPsBCsWwz_BvbZZHIVQ6wSZK

4. Cross Validation, Sampling, train test split in Machine Learning
https://www.youtube.com/playlist?list=PLE-8p-CwnFPsHtol5WXHhq_B3kQPggHH2

5. Drop and Delete Operations in Python Pandas
https://www.youtube.com/playlist?list=PLE-8p-CwnFPtvqVVK7QVFsMvDvp2YgCnR

6. Matrices and Vectors with python
https://www.youtube.com/playlist?list=PLE-8p-CwnFPsndwnZnL7nXW5mIrdRmgdg

7. Detect Outliers in Machine Learning
https://www.youtube.com/playlist?list=PLE-8p-CwnFPvyCX35yES5D9W7vThiUzwk

8. TimeSeries preprocessing in Machine Learning
https://www.youtube.com/playlist?list=PLE-8p-CwnFPv10bru3719xzDNIgbO6hXA

9. Handling Missing Values in Machine Learning
https://www.youtube.com/playlist?list=PLE-8p-CwnFPvOec0LZ40Bt8OQcbLFa236

10. Dummy Encoding Encoding in Machine Learning
https://www.youtube.com/playlist?list=PLE-8p-CwnFPvu7YriqMZsL9UDbqUUk90x

11. Data Visualisation with Python, Seaborn, Matplotlib
https://www.youtube.com/playlist?list=PLE-8p-CwnFPuYBYsmbfMjROOCzKjCwyMH

12. Feature Scaling in Machine Learning
https://www.youtube.com/playlist?list=PLE-8p-CwnFPtwpVV3FwzwYZYR5hT3i52G

13. Python 3 basics for Beginner
https://www.youtube.com/playlist?list=PLE-8p-CwnFPu-jseUMtc4i47jQZN4PNbf

14. Statistics with Python
https://www.youtube.com/playlist?list=PLE-8p-CwnFPta0COlxS6E5u14m5ouzbRU

15. Sklearn Scikit Learn Machine Learning
https://www.youtube.com/playlist?list=PLE-8p-CwnFPtAGb29r8F7up9ilZUXt3l1

16. Python Pandas Dataframe Operations
https://www.youtube.com/playlist?list=PLE-8p-CwnFPv_63lkT_Tztiwknv_zGTNy

17. Linear Regression, Supervised Machine Learning
https://www.youtube.com/playlist?list=PLE-8p-CwnFPslDi6sfFbFK4KXcVlLwaOM

18 Interiew Questions
https://www.youtube.com/playlist?list=PLE-8p-CwnFPt7VBhcnh82y0autSzuOrZp

19. Jupyter Notebook Operations
https://www.youtube.com/playlist?list=PLE-8p-CwnFPtqkFd67OZcoSv4BAI7ez5_